Reading papers and locating the evidence for claims

Why it matters

Research interviews and thesis conversations often start with "tell me about a paper you liked" and quickly turn into "what is the weakest part of its evaluation?". Engineering needs the same skill for design documents, benchmarks and vendor claims: separating what a document asserts from what it demonstrates.

Core ideas

Read in passes. Keshav's three-pass method is a good default. A first pass of five to ten minutes (title, abstract, introduction, headings, conclusion, references) tells you the category of paper, its context, whether the assumptions look valid, the claimed contributions and whether it is clearly written. The second pass, up to about an hour, reads the body and figures carefully while skipping proofs and fine detail. The third pass reconstructs the work as if you had to re-implement it, which is where hidden assumptions and weak arguments surface. Most papers deserve two passes; papers you will discuss in an interview or build on deserve three.

Structured notes. For every paper, record:

• Problem: what fails today, for whom, and how much it matters.
• Assumptions: failure model, hardware, workload or data, threat model, scale.
• Mechanism: the key idea in two sentences, then the parts that make it work.
• Contribution: what is new relative to the closest work.
• Evidence: what actually supports each claim.
• Baselines: what was compared, and whether it was configured and tuned fairly.
• Ablations: which components were removed to show they matter.
• Limitations: stated and unstated.
• Implementation: size, language, simplifications.
• Follow-up: the experiment you would run next.

Claim-evidence tables. Put each claim from the abstract and introduction in a row. Record its type (correctness guarantee, measurement, qualitative or design argument), where the evidence is (section, figure, proof, appendix), the conditions under which it holds, and a rating such as strong, partial or missing. An abstract may promise "up to 3x faster"; the table forces you to find the figure and see whether that is a median, a best case or one workload. Some claims have no evidence at all, and finding them is the point.

The Raft paper (extended version) is a good first target because different claims rest on different kinds of evidence: understandability on a user study (Section 9.1), safety on a formal specification and proofs (Section 9.2), and leader-election behaviour on measurements (Section 9.3). One row of such a table might read:

• Claim: randomized election timeouts let the cluster replace a crashed leader quickly.
• Type: measurement.
• Evidence: Section 9.3, repeated leader crashes on a five-server cluster.
• Conditions: the authors' cluster, network and timeout ranges, with a deliberately adversarial setup.
• Strength and open questions: convincing for that setup; a reader should still ask how it behaves with wide-area latencies or overloaded servers.

Read the evaluation like a reviewer. For each figure ask: what are the axes and units, what is normalized to what, how many runs, what do the error bars mean, which baseline version and configuration, which workload, and what the plot would look like if the idea did not work. Roscoe's notes on reviewing for systems conferences show the reviewer's checklist and tone.

Generate alternatives. Before accepting a design, ask what the simplest alternative would be (a bigger cache, a different timeout, a stronger baseline, retrieval instead of fine-tuning) and whether the evaluation rules it out.

Common misconceptions

• "One careful front-to-back read is enough." A single linear pass leaves you with the authors' framing; purposeful passes let you test it.
• "Published at a top venue means the claims are established." Review checks plausibility under time pressure; it rarely re-runs experiments.
• "The abstract summarizes the evidence." It summarizes the claims; the evidence lives in specific figures, proofs and appendices, often with narrower conditions.
• "Critique means listing weaknesses." Useful critique is specific, acknowledges strengths and proposes the experiment that would settle the doubt.
• "I must understand every detail before discussing a paper." You need the claims, mechanism and evidence; say which details you skipped.

How to approach it

Do one paper deeply rather than skimming ten: build the Raft claim-evidence table, then answer some of the MIT 6.5840 per-paper questions before reading the course's notes. Keep notes in the app's Research notes so every paper gets the same fields. Graduates are expected to summarize accurately and name one real limitation; stronger candidates explain why the authors made each design choice and which experiment they would add.