API Latency Incident — Postmortem
A one-line config typo in the v2.31 deploy capped the database connection pool at 10 connections. The pool saturated under normal traffic, queueing requests until checkout calls timed out. Rolled back to v2.30; latency returned to baseline.
Impact
Duration
09:12 → 10:48 UTC · detection took 22m
p99 latency peak
Baseline 180ms · 23× normal
Est. lost orders
14% of checkout requests timed out
Users affected
Saw errors or >1s checkout
Customer impact was concentrated in checkout. Checkout makes three database queries per request, so it exhausted the starved pool first — these were the requests that timed out and the orders we lost.
Timeline
All times UTC. Click any entry to expand the detail. Detection lagged the deploy by 22 minutes; rollback was the fix.
Latency Curve
p99 spiked within minutes of the 09:12 deploy, breached the 1s SLO by 09:20, and only fell back to the 180ms baseline (dashed line) after the 10:15 rollback. Hover the curve for exact values.
| Time (UTC) | p99 latency | Phase |
|---|
Root Cause — Causal Chain
A single mistyped number propagated into customer-visible timeouts. Each step caused the next:
Config typo lowered the pool cap
v2.31 set the database pool max-connections from 100 → 10 — a typo in the config, not an intended change.
Pool saturated under normal traffic
No traffic spike was needed. Ten connections could not cover steady-state concurrency, so the pool ran fully checked-out almost immediately.
Requests queued waiting for a connection
With every connection busy, new requests blocked in the pool's wait queue until one freed up, instead of failing fast.
Queue time surfaced as request latency
Time spent waiting for a connection is counted inside the request, so p99 climbed from 180ms toward 4.2s as queue depth grew.
Checkout timed out first
Checkout runs three queries per request, so it accumulated the most queue time and crossed the timeout threshold before lighter endpoints — which is why 14% of checkouts failed.
Action Items
Sort by any column or search to find an item. The config revert is shipped; guardrails to prevent a repeat are tracked below.
Lessons Learned
A one-line typo had outsized impact
Changing a single number — 100 to 10 — degraded the whole API and cost ~$23k in orders. Small config edits are not low-risk by default.
Guardrails beat review for resource limits
Code review did not catch the change. Resource-limit edits need machine guardrails — diff gates, saturation alerts, staging load tests — not just a human glance.