Failure injection and prototype evaluation

Why it matters

A fault-tolerance claim is a claim about behaviour under failure, so steady-state benchmarks cannot support it. Independent analyses of widely used databases and queues have repeatedly found lost writes, stale reads and split-brain behaviour under partitions in systems whose documentation promised stronger guarantees. Systems research reviewers therefore expect evidence under injected faults, and practitioners use the same techniques to find bugs before users do. In a research interview, being able to design such an experiment for your own prototype is a strong signal.

Core ideas

Choose faults from the failure model. Crash and restart processes, with and without losing unsynced data; pause them (a stopped process imitates a long garbage-collection pause); partition the network completely, partially (some nodes still see each other) and asymmetrically; add latency and loss; skew clocks; and inject disk errors or corruption. Failure studies are a good guide: the majority of catastrophic failures in widely used distributed data systems came from mishandling errors the software had already detected, most network-partition failures were catastrophic and many could be triggered by isolating a single node, and a single file-system fault on one replica can cause data loss or unavailability despite redundancy.

The Jepsen pattern. Several clients issue concurrent operations from a generator and record every invocation and completion, including timeouts whose outcome is unknown. A nemesis injects faults on a schedule. Afterwards a checker tests whether the history could have come from the claimed model: Knossos or Porcupine for linearizable registers and maps, Elle for transactional isolation anomalies. Indeterminate operations must be treated as possibly applied; ignoring them hides real anomalies or invents false ones.

Chaos experiments. In production or production-like environments, state a hypothesis about an observable steady-state metric such as successful requests per second, introduce a real-world event such as an instance failure, and look for deviations while limiting the blast radius and defining abort conditions. Chaos experiments test the whole socio-technical system (alerts, fallbacks, retries, operators), not one protocol's safety.

Deterministic simulation. Run the distributed system in one process with simulated network, disk and time, driven by a pseudorandom seed. Faults and message reorderings become cheap to explore, and any failure replays exactly from its seed. The cost is architectural: every source of nondeterminism (threads, clocks, I/O) must go through controllable interfaces.

Evaluating a prototype. Report what happens during and after faults: throughput and latency over time with fault windows marked, unavailability duration, time to recover or elect a new leader, data loss or anomalies found by the checker, and how results vary across repeated runs. Compare with baselines under the same fault schedule. State what was not tested; a clean history under one schedule is evidence for those faults, not a proof of correctness.

Common misconceptions

• "No anomalies found means the system is correct." Checkers only see histories produced by the faults and workloads you ran; coverage matters as much as the checker.
• "Timed-out operations can be counted as failures." A timed-out write may have been applied, so histories must treat it as indeterminate.
• "Chaos engineering means randomly killing things in production." It is hypothesis-driven experimentation with controlled scope and abort criteria.
• "Killing processes covers crash faults." Many bugs need crashes at specific points, such as between writing data and syncing it, or during recovery itself.
• "Fault-tolerance results can be reported like throughput results." They need time series around fault events and recovery metrics, not only run-wide averages.

How to approach it

Read a few Jepsen analyses to absorb the method, then reproduce it at small scale: record histories from your Raft-based key/value store or another small replicated service under partitions and check them with Porcupine. Turn the findings of one failure study into three concrete test cases. In interviews, describe the fault model, workload, checker and metrics before naming tools.