Retries, deadlines, overload and graceful degradation

Why it matters

A dependency slows down, callers time out and retry, the extra traffic slows the dependency further, queues grow, and soon services that never had a bug are failing too. That feedback loop, more than the original fault, is behind many long outages. Retries and overload come up in interviews because the naive answers (retry three times, add a bigger queue) make things worse, and because every design discussion eventually reaches "what happens when this dependency is slow?"

Core ideas

Timeouts and deadlines. Every remote call needs a timeout chosen from the dependency's latency distribution (for example somewhat above its p99.9), not a round number. Better still, the edge gives each request a deadline and passes the remaining budget downstream, so no service starts work its caller has already abandoned; servers check the deadline before expensive steps and drop requests that are already late.

Retries amplify. If each of three layers makes up to three attempts, one user request can become 27 calls at the bottom layer, exactly when it is struggling. Rules of thumb:

• Retry only idempotent or deduplicated operations, and only errors likely to be transient.
• Retry at one layer (usually the one with the most context), not at every layer.
• Use capped exponential backoff with jitter, so clients that failed together do not retry together.
• Bound retries with a budget, for example allowing retries only up to a small fraction of recent successful requests, implemented as a token bucket. While the dependency is healthy, retries flow freely; while it is failing, they stop adding load.

sleep = random_between(0, min(cap, base * 2 ** attempt))      # "full jitter"
retry only if error.retryable and budget.try_take() and deadline_remaining > sleep

Overload and goodput. When offered load exceeds capacity, queues grow, latency rises past client timeouts, and the server spends its capacity on requests whose callers have already left. Throughput can stay high while goodput (useful completed work) collapses. The cure is to refuse work early and cheaply:

• Admission control and load shedding: reject excess requests at the front door with a clear, retry-friendly error, preferably by priority (drop prefetch and batch traffic before user-facing requests) and per tenant.
• Bounded queues and backpressure: a full queue should push back on producers (block, reject or slow them) instead of growing. Under backlog, serving newest work first or discarding requests already past their deadline keeps the system useful.

Circuit breakers. A client-side breaker tracks recent failures to a dependency; past a threshold it opens and fails fast, then admits a few trial requests after a cool-down (half-open) to test recovery. It protects the dependency and frees the caller's threads, but each client sees only part of the picture, so breakers can flap.

Graceful degradation. Decide in advance what the product does without each dependency: serve cached or stale data, hide a recommendation panel, accept writes into a queue for later. Fallbacks that run only during incidents are rarely tested and may overload something else; prefer degraded modes that are exercised regularly.

Cascading and metastable failures. Overload spreads through shared dependencies and retries. In a metastable failure the trigger (a short spike, a cache flush) disappears, yet the system stays down because the recovery load (retries, cold caches) keeps it overloaded. Recovery then requires shedding load or blocking retries, not only fixing the trigger.

Common misconceptions

• "Retries improve reliability." They hide rare transient errors; under sustained failure they multiply load. Budgets and single-layer retries keep the benefit without the amplification.
• "Exponential backoff alone prevents retry storms." Without jitter, clients that failed at the same moment stay synchronized.
• "A bigger queue absorbs overload." It converts overload into latency, and requests then time out after consuming resources; bound queues and shed load.
• "Generous timeouts avoid errors." Long timeouts tie up threads and connections during an incident; base them on measured latency and the caller's deadline.
• "A circuit breaker fixes overload." It limits the damage one client does; the server still needs its own admission control.

How to approach it

Read the Builders' Library articles on timeouts and on load shedding, then SRE book Ch. 21 and 22. Build intuition with a small experiment: a server with fixed capacity and a load generator whose clients retry; plot goodput as offered load passes capacity, with and without shedding, jitter and a retry budget. Graduates should know timeouts, backoff with jitter and idempotent retries. Mid-level candidates should design deadlines, budgets, shedding and degraded modes for a given call graph. Senior candidates should reason about amplification, metastability and per-tenant fairness, and explain how they would test overload behavior before production.