Research coding: prototypes, reference checks and instrumentation

Why it matters

A bug in research code becomes a false claim. Unlike product code, there is often no user to notice that an output is wrong; the only defence is checks built into the code. Research-engineering interviews therefore probe testing strategy, numerical care and how you would trust your implementation of a paper.

Core ideas

Prototype for change. Research code changes daily, so optimize for reading and modifying: small pure functions for the core computation, configuration in files rather than constants scattered through the code, one entry point per experiment, notebooks for exploration but scripts for anything that produces a reported number. Skip production concerns such as polish and scalability unless the research needs them, but never skip correctness checks.

Reference implementations and differential testing. For every optimized or intricate component, keep a slow version you can verify by reading it: naive attention beside a fused kernel, a sequential MapReduce beside the distributed one (MIT 6.5840 Lab 1 supplies such a sequential version), brute-force search beside an index, an exact percentile beside a streaming sketch. Generate many random and edge-case inputs and compare outputs. Csmith applied this idea at scale, finding compiler bugs by compiling generated programs with several compilers and comparing results. Property-based testing tools such as Hypothesis generate inputs and shrink failures to minimal examples.

Tolerances are part of the specification. Floating-point results change with summation order, fusion and precision. Compare with explicit absolute and relative tolerances derived from the precision in use, and investigate failures instead of loosening tolerances until tests pass.

Numerical traps. Compute softmax and log-likelihoods with max subtraction or log-sum-exp. Watch for overflow and underflow in half precision, cancellation when subtracting nearly equal numbers, counter overflow and silent dtype promotion. For ML code, check gradients against finite differences on tiny inputs, as the CS231n notes describe, and confirm that a model can overfit a tiny batch before scaling up.

Measurement code is research code. Use monotonic clocks with adequate resolution, exclude warm-up explicitly, remember that percentiles depend on the interpolation method and sample count, and keep units straight (milliseconds versus seconds, bits versus bytes, MB versus MiB). Test the measurement path on synthetic inputs with known answers.

Instrument everything. Each run writes its resolved configuration, commit and diff, environment, seeds and raw measurements as structured records (for example JSON lines) into its own directory, starting at launch so crashed runs remain traceable. This makes later analysis and reproduction possible.

Iteration speed. Fast feedback beats raw speed: tiny configurations for debugging, cached preprocessing, parallel parameter sweeps and early validation of configurations. Check periodically that small proxies rank variants the same way full runs do; a misleading proxy is worse than a slow experiment.

Tests at the right level. Unit tests on core math; property and metamorphic tests (permuting independent inputs should not change an aggregate, doubling every capacity should not reduce throughput); regression tests against stored small outputs; and an end-to-end smoke run that exercises the whole pipeline in under a minute.

Common misconceptions

• "Research code does not need tests." It needs different tests: fewer on interfaces, more on numerical and measurement correctness.
• "The loss goes down, so the implementation is right." Many bugs still let training improve; compare against references and sanity checks.
• "Bit-exact equality is the right comparison." For floating point it usually is not; choose tolerances deliberately.
• "I will add logging once results look interesting." By then the configurations of earlier runs are gone.
• "Notebooks are fine for final results." Hidden state and out-of-order execution make reported numbers hard to trust.

How to approach it

Take one optimized function from your own work or a course lab, write a reference and a differential test with generated inputs, then add a run harness that logs configurations and raw data. In interviews, be ready to explain how you would verify an implementation of a paper's algorithm before trusting its numbers; graduates should propose reference checks and tests, senior candidates should also discuss harness design and the cost of each check.