Failure models, partial failure and timeouts

Why it matters

On one machine, a component usually either works or the whole program dies. In a distributed system, parts fail independently: one replica crashes, a switch drops a fraction of packets, a process freezes for seconds during garbage collection, a disk becomes slow without dying. The caller observes only silence or delay, and silence is ambiguous. Nearly every hard question in this domain (is it safe to retry, who is the leader, do I still hold this lock) reduces to that ambiguity, which is why failure reasoning is where memorized architectures and defensible designs part ways.

Core ideas

Fault models say what a failed component may do:

• Crash-stop: the node halts and never returns. Easiest to reason about, rarely the whole truth.
• Crash-recovery: the node halts and restarts later with only what it had made durable. Protocols must therefore state what is persisted before acknowledging; a vote or log entry kept only in memory can be forgotten and then contradicted.
• Omission: messages are lost while the node keeps running. Retransmission hides this at the cost of duplicates and reordering.
• Byzantine: a node may behave arbitrarily, including sending conflicting messages. Classic protocols need 3f+1 replicas to tolerate f such nodes. Typical datacenter systems assume non-Byzantine nodes and handle corruption with checksums, authentication and validation instead; awareness is enough here.

Timing models matter as much. In an asynchronous model nothing bounds message delay or processing time, so a slow node cannot be told apart from a crashed one, and the FLP result shows that no deterministic consensus protocol can guarantee termination if even one process may crash. Practical systems assume partial synchrony: bounds hold most of the time, so they keep safety unconditional and make progress only while the network behaves.

A timeout is a guess. When a request times out, it may have been lost, it may have executed with the reply lost, or it may still be running and take effect later:

These rows look identical to the caller, so a retry is safe only for idempotent or deduplicated operations, and the honest result of an API call is sometimes "unknown".

Partitions split nodes into groups that cannot communicate, possibly asymmetrically (A hears B, B does not hear A). Each side may conclude the other is dead. If both sides keep acting as leader or lock holder you get split brain; requiring a majority keeps one side safe and makes the other unavailable.

Detection is a tradeoff. Short timeouts detect crashes quickly but suspect healthy nodes during load spikes, triggering failovers that add more load. Long timeouts avoid false suspicion but leave requests stuck. Derive timeouts from measured latency (a high percentile plus margin), and make every action taken on suspicion safe even if the suspicion is wrong.

Gray failure is the common middle ground: a node passes health checks but fails or crawls on some requests. Fail-slow components often hurt more than crashed ones because nothing removes them automatically.

Common misconceptions

• "A timeout means the operation failed." It means the caller stopped waiting. The operation may have completed; treat the outcome as unknown until confirmed.
• "TCP makes the network reliable." TCP retransmits and orders bytes within one connection. It cannot tell you whether the remote application processed a request before the connection broke, and it does not survive process crashes.
• "Crash-recovery is just crash-stop plus a restart." A restarted node that forgot an in-memory promise (a vote, a lease, an acknowledged write) can violate safety. What is flushed to disk before replying is part of the protocol.
• "Partitions are rare, so handle them later." Partial connectivity, overloaded links and long pauses produce the same symptoms routinely; the design must say what each side does while it cannot reach the others.
• "We can detect failures accurately." In an asynchronous network no detector is both always accurate and always complete. Detectors are tuned, and protocols must stay correct under false suspicion.

How to approach it

Start with DDIA's chapter on the trouble with distributed systems, then make it concrete: for one request path you know (a payment, a booking, sending an outreach message) write the failure-mode table and decide what the client must do for each row. Graduates are expected to recognize partial failure and the ambiguity of timeouts. Mid-level candidates should raise failure handling unprompted and propose idempotency or fencing where needed. Senior candidates should name the assumptions (fault model, timing, clocks) under which a design is safe and say what happens when each is violated. In code, the MapReduce lab trains recovery by re-execution and the key/value lab makes you return "maybe" for a retransmitted write.