Critical

DB Connection Pool Exhaustion

When your database connection pool fills up, queries queue and apps hang. OnCallReady identifies idle or leaked connections, terminates them, rebalances the pool, and restarts offending services — typically under 25 seconds.

Avg Resolution
22s
Severity
Critical
Success Rate
97%
Humans Paged
0

Trigger Conditions

/db.*(pool|connection|exhausted|timeout|max.conn|too.many)/i

Fires on database connection-related alerts indicating pool saturation or connection leak. Typical triggers: "DB connection pool exhausted on api-server", "Too many connections to PostgreSQL", "Connection timeout after 30s", "Max connections reached (100/100)".

What the Agent Does

1

Query active connections

Runs pg_stat_activity (or equivalent) to enumerate current connections by state, duration, and application. Identifies idle connections, long-running transactions, and potential connection leaks.

2

Terminate idle connections

Issues pg_terminate_backend for connections idle >5 minutes. Preserves active query connections. Frees pool slots immediately without requiring service restarts.

3

Identify and restart connection-leaking services

Cross-references application tags on connections to identify which service is leaking. Performs a rolling restart of that service to reset its internal connection pool state.

4

Verify pool availability

Checks connection count drops below 70% of maximum. Confirms application health endpoints return 200 and database queries complete normally.

5

Log and optionally right-size pool limits

Records freed connection count, offending service, and incident timeline. Flags if the max_connections setting is consistently approached — surfaces this for infrastructure review.

Example Incident Log

incident-5201 · db-connection-pool · prod-db-primary
[11:44:18] ALERT DB connection pool exhausted (100/100) on prod-db-primary
[11:44:18] Matched runbook: db-connection-pool
[11:44:19] Querying pg_stat_activity...
[11:44:20] 62 connections idle >5min · application=api-worker
[11:44:20] 38 connections active · normal query patterns
[11:44:21] Terminating 62 idle connections via pg_terminate_backend
[11:44:22] Freed 62 connections · Pool: 100/100 → 38/100
[11:44:22] Connection leak source: api-worker (v2.3.1) — no pool.release() calls
[11:44:23] Rolling restart of api-worker (2 instances)
[11:44:29] api-worker restarted · Health checks passing
[11:44:30] Pool stable at 24/100 connections
[11:44:40] ✓ RESOLVED Pool: 100/100 → 24/100 · Duration: 22s
[11:44:40] On-call: undisturbed. Connection leak flagged for dev review.

Fix DB pool exhaustion before queries start timing out

OnCallReady catches connection leaks and fixes them in seconds. See it in action.