From interest to falsifiable research question

Why it matters

Many stalled research projects fail before the first experiment: the "question" was a direction ("look at serverless cold starts") rather than something an experiment can answer. Supervisors and research interviewers test this quickly. They ask what you would measure, against what, and which result would change your mind. A crisp question also protects against a common thesis failure: months of building without a claim to evaluate.

Core ideas

Area, problem, question, hypothesis. An area is a field (cloud resource management). A problem is a gap with consequences (co-located batch jobs inflate the tail latency of interactive services). A question asks something answerable about it (which shared resource explains most of the inflation?). A hypothesis commits to an answer that data could contradict.

Operationalize every word. A falsifiable hypothesis names the system or model, the workload or data, the conditions, the metric, the comparison and the size of effect that matters. "Faster", "scalable" and "robust" must become numbers with units and a stated statistic (median, p99, mean over seeds).

Illustrative rewrites (hypotheses to test, not known results):

• Cloud: on a two-socket server at 70% of peak load, a memory-bandwidth-heavy co-runner raises a key-value cache's p99 latency more than a cache-thrashing co-runner with equal CPU use. Refuted if the ordering reverses or the gap stays within run-to-run noise.
• Networking: for flows under 100 KB with 16 synchronized senders behind a shallow-buffer switch, pacing every sender lowers p99 flow completion time without reducing median goodput by more than 5%.
• Databases: on join-heavy queries with correlated predicates, estimator X reduces the number of plans that run at least twice as slow as the best known plan, without slowing single-table queries.
• Systems: group commit raises write-ahead-log throughput on NVMe only once concurrency exceeds a threshold; below it, commit latency increases.
• ML and agents: at an equal token budget, an agent that verifies its own tool outputs completes more tasks than the same agent sampling twice and keeping the better attempt.

The last example shows why the comparison is part of the question: without a matched-compute baseline, a verifier could win simply by spending more tokens.

Question types need different evidence. Descriptive questions (what does this workload look like?) need careful measurement and sampling. Comparative or causal questions (does X improve Y?) need controlled experiments. Design questions (can a system achieve property P under constraint C?) need a prototype plus evidence that P holds where it matters. Mechanism questions (why does X happen?) need experiments that isolate a cause, often by removing it.

Falsifiability. A claim is useful when it forbids outcomes. "Our scheduler can improve utilization" forbids nothing; "it raises mean CPU utilization by at least 10 points on trace T without increasing p99 job delay" does. Write the refuting result down before running anything.

Importance and feasibility. Keshav's shorthand, research should be "worth doing, doable, and not done", is a useful filter. The Heilmeier questions (what are you trying to do, how is it done today, what is new, who cares, risks, cost, duration, mid-term and final exams) force you to state the goal without jargon and define success checks in advance.

Common misconceptions

• "A topic is a research question." "LLM agents for outreach" is an area. A question needs a measurable outcome and a comparison.
• "Falsifiable means I expect to be wrong." It means a specific observation would show you wrong; you can still be confident.
• "Bigger questions make better theses." A narrow question answered convincingly beats a broad one answered anecdotally. Scope to the time, data and hardware you will actually have.
• "I will state the hypothesis after the first results." Fine for exploration, but a hypothesis shaped by the data has not been tested yet; confirm it on fresh runs, workloads or data.
• "Building the system is the research." A prototype is a means. The contribution is what it shows: a property achieved, a tradeoff exposed or a mechanism explained.

How to approach it

Take one real interest, ideally from your own projects or coursework, and write three hypotheses with the template in setting S, under workload W, change X moves metric M relative to baseline B by at least D; refuted if .... Answer the Heilmeier questions for the strongest. Expect follow-ups such as "how would you know you are wrong?", "what is the simplest baseline?" and "what would you do in the first two weeks?". At graduate level a precise question and a credible first experiment suffice; senior candidates should also argue why the question matters now and how far the answer generalizes.