SLOs, incidents, capacity and tail latency

Why it matters

"Make it reliable" is not a requirement; "99.9% of checkout requests succeed within 400 ms over 28 days" is. SRE practice provides the language to set such targets, decide when to slow releases, run incidents calmly and plan capacity before the traffic arrives. SRE and infrastructure interviews test the vocabulary and, more importantly, the judgment behind it.

Core ideas

SLI, SLO, SLA, error budget. An SLI is a measured ratio of good events to valid events (successful requests, requests faster than a threshold, data fresher than a bound). An SLO is a target for that ratio over a window, such as 99.9% over 28 days. An SLA is a contractual promise, usually looser, with consequences attached. The error budget is the allowed shortfall: at 99.9%, one request in a thousand may fail. Measure as close to the user as possible, and use percentiles or threshold ratios rather than averages, which hide the slow tail users notice.

Error budget policy. While budget remains, ship faster and take risks; once it is spent, prioritize reliability work and slow risky changes. Agreeing on the policy in advance turns arguments into decisions.

Alerting on burn rate. Burn rate is how fast you consume the budget relative to the window: at a burn rate of 1 the budget runs out exactly at the window's end. Page on fast burns confirmed over two windows (a long one for significance, a short one so the alert clears soon after recovery) and open tickets for slow burns. Alert on symptoms users feel; use cause metrics (saturation, queue depth) for diagnosis.

Incident response. Separate roles (incident commander, operations, communications), keep a running log, and mitigate first: roll back the last change, drain a bad zone, shed load or fail over before understanding the root cause. Afterwards write a blameless postmortem about contributing factors, with owned action items. For diagnosis, the USE method (utilization, saturation and errors for every resource) and the golden signals (latency, traffic, errors, saturation) keep investigations systematic.

Capacity planning. Start from peak demand and its growth, not the average. Divide by per-instance throughput measured at the latency target in load tests, add headroom for spikes and for losing failure domains (N+2 survives one planned and one unplanned loss), and check concurrency with Little's law: requests in flight equal arrival rate times latency, so 2,000 requests per second at 50 ms means about 100 concurrent requests.

Tail latency. When a request fans out and waits for every leaf, its latency is the maximum of theirs. If each leaf is slower than its own p99 with probability 1%, a request touching 100 leaves is slowed with probability 1 - 0.99^100, about 63%. Remedies: reduce fan-out, send hedged requests (a second copy after a delay near the p95, keep whichever answers first, cancel the other), use tied requests that cancel on dequeue, and partition finely so slow work can be redistributed. Hedging trades a few percent of extra load for a much shorter tail, so it must be capped and the request idempotent.

Common misconceptions

• "A higher SLO is always better." Each extra nine costs disproportionately more and users often cannot tell; derive the target from user needs and dependency limits.
• "Average latency is a good SLI." Averages hide the slow minority; use percentiles or the fraction of requests under a threshold.
• "During an incident, find the root cause first." Restore service first; investigation continues afterwards with evidence preserved.
• "Capacity is average load divided by per-server throughput." Plan for peaks, latency targets, failure-domain loss and growth.
• "Hedging doubles the load." Sending the backup only after a high-percentile delay adds little load while cutting the tail; it still needs a cap.

How to approach it

Read SRE book Ch. 4 and Workbook Ch. 2 and 5, then write an SLO document for a service you know. Read "The Tail at Scale" and redo its fan-out arithmetic yourself. Practice incident narration: for a public postmortem you have read, say what you would have mitigated first and why. Graduates should define the terms and compute an error budget. Mid-level candidates should design SLIs, alerts and a capacity estimate. Senior candidates should negotiate SLO targets, design the error budget policy and connect tail latency to architecture choices.