How OnCallReady resolves real incidents —
in seconds, with no human in the loop.

"The latency spike pattern — sudden onset, all outbound traffic affected, no change in inbound traffic — pointed to an egress routing problem rather than an application regression. I checked the deploy log from 2:31am and saw the egress rule change introduced in PR #4187. The diff confirmed a new NAT gateway route that wasn't wired into the transit gateway yet. This is a known misconfiguration class — the fix is a targeted Terraform revert, not a full service rollback. I confirmed impact scope across downstream services first to make sure nothing worse was happening before I applied the change. Recovery validated within 5s of apply completing."

"Linear heap growth at a constant rate rules out a traffic spike — if it were traffic-driven, growth would correlate with request volume. This is a classic listener leak: something is being registered on an EventEmitter per operation without cleanup. The diff on payment-worker-v3 showed 47 new lines in processQueue() and EventEmitter instantiation inside the job handler with no corresponding removeListener. I didn't roll back the deploy because the service was still processing payments successfully — the right move was stabilize first, fix second. Scaling to 6 replicas bought time. The PR is the permanent fix; an engineer can review and merge when they wake up."

"The cert expired 24 seconds before I received the alert — the synthetic monitor caught it in the first check after expiry. I confirmed independently with curl before touching anything; synthetic monitors can have false positives. The curl output was unambiguous: certificate expired, exact timestamp. I checked customer error logs first — zero SSL errors had surfaced in customer traffic, likely because the expiry hit at 4am during the low-traffic window. That gave me about 45 minutes before US-East business hours traffic would hit this hard. The Vault path was pre-configured in the runbook. Pull, issue, apply, validate chain — this is a deterministic procedure. I didn't need to wake anyone up."