When your database connection pool fills up, queries queue and apps hang. OnCallReady identifies idle or leaked connections, terminates them, rebalances the pool, and restarts offending services — typically under 25 seconds.
Fires on database connection-related alerts indicating pool saturation or connection leak. Typical triggers: "DB connection pool exhausted on api-server", "Too many connections to PostgreSQL", "Connection timeout after 30s", "Max connections reached (100/100)".
Runs pg_stat_activity (or equivalent) to enumerate current connections by state, duration, and application. Identifies idle connections, long-running transactions, and potential connection leaks.
Issues pg_terminate_backend for connections idle >5 minutes. Preserves active query connections. Frees pool slots immediately without requiring service restarts.
Cross-references application tags on connections to identify which service is leaking. Performs a rolling restart of that service to reset its internal connection pool state.
Checks connection count drops below 70% of maximum. Confirms application health endpoints return 200 and database queries complete normally.
Records freed connection count, offending service, and incident timeline. Flags if the max_connections setting is consistently approached — surfaces this for infrastructure review.
DB connection pool exhausted (100/100) on prod-db-primaryCRITICAL: Too many connections to PostgreSQL on prod-pg-1Datadog: postgresql.connections > 95% of max for >2m on prod-pg-1PagerDuty: Connection timeout after 30s — pool saturationApp: HikariCP CannotGetConnectionException — pool exhaustedpg_stat_activity: many idle-in-transaction > 5min, app=api-worker1. SELECT count(*), state FROM pg_stat_activity GROUP BY state ├── Most connections idle-in-transaction > 5min → app leak — caller did not COMMIT/ROLLBACK ├── Most connections idle → pool sizing too aggressive per app instance └── Most connections active long-running → slow query blocking the rest 2. SELECT application_name, count(*) FROM pg_stat_activity GROUP BY application_name ORDER BY count(*) DESC LIMIT 5 └── Identify the leaking service 3. Decide: ├── Idle leak → pg_terminate_backend(pid) for stale sessions ├── Slow query → pg_cancel_backend(pid) then optimize └── Repeated leak from one service → rolling restart that service
# 1. Inspect active connections psql -h prod-db-primary -U admin -d app -c " SELECT pid, usename, application_name, state, query_start, xact_start FROM pg_stat_activity ORDER BY xact_start NULLS LAST LIMIT 20;" # 2. Terminate idle-in-transaction connections older than 5 minutes psql -h prod-db-primary -U admin -d app -c " SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state IN ('idle', 'idle in transaction') AND (now() - state_change) > interval '5 minutes' AND pid <> pg_backend_pid();" # 3. Cross-reference to find the leaking service psql -h prod-db-primary -U admin -d app -c " SELECT application_name, count(*) FROM pg_stat_activity GROUP BY application_name ORDER BY count(*) DESC;" # 4. Rolling restart the leaking service kubectl rollout restart deploy/api-worker # 5. Verify pool is back below 70% capacity psql -h prod-db-primary -U admin -d app -t -c " SELECT count(*) FROM pg_stat_activity;"
Our agent queries pg_stat_activity, bucketing sessions by state and application_name, terminates all idle-in-transaction connections longer than 5 minutes, then cross-references to identify the leaking service. It performs a rolling restart via kubectl rollout restart and verifies pool usage drops below 70% before resolving.
Stop connection-pool exhaustions before your queries start timing out.