Reproducibility, provenance and artifacts

Why it matters

A result you cannot regenerate is a result you cannot defend. When a reviewer, supervisor or interviewer asks "what exactly produced this number?", "probably the run from Tuesday" ends the discussion. Provenance also protects you: many surprising results turn out to be configuration mistakes, and you can only find them if the configuration was recorded.

Core ideas

Three words, carefully. ACM's badging policy (version 1.1) defines repeatability as the same team obtaining a result again with the same setup, reproducibility as a different team obtaining it with the original authors' artifacts, and replicability as a different team obtaining it with independently developed artifacts. ACM swapped the last two terms in 2020 to align with other sciences, and some communities still use them the other way round, so define the terms whenever you use them.

Provenance for every run. Record automatically:

• code: commit hash plus the uncommitted diff, or refuse to run on a dirty tree;
• the fully resolved configuration, including defaults;
• data: name, version or content hash, split and preprocessing version;
• environment: OS and kernel, library versions, compiler flags, container image digest;
• hardware: CPU or GPU model, memory, cloud instance type and region, frequency-scaling and SMT settings;
• seeds, command line, host, start and end time;
• raw outputs, not only aggregates.

Sandve et al.'s Ten Simple Rules for Reproducible Computational Research compress the habits: track how every result was produced, avoid manual data manipulation, archive exact versions of external programs, version-control scripts, record intermediate results, note seeds, keep the raw data behind plots and connect statements in the text to underlying results.

Harness, not heroics. Give every run an identifier and a directory holding its configuration, environment snapshot, logs and results. Figures come from scripts that read those directories, never from numbers copied by hand. A thesis results table should be regenerable with one command.

Nondeterminism. Seeds make sampling repeatable, but GPU kernels, parallel reductions, thread scheduling and remote services (including hosted language models) can still vary. For each source, decide whether it matters for your claim: control it (deterministic algorithms, pinned model versions, cached API responses) or measure it with repeated runs. Deterministic modes can cost speed, so record whether they were on.

Artifacts and badges. Many systems conferences run an artifact evaluation, typically after acceptance. ACM's badges mark artifacts that are archived publicly (Available), checked to be documented, consistent, complete and exercisable (Evaluated – Functional) or well structured for reuse (Evaluated – Reusable), and results obtained by others with (Reproduced) or without (Replicated) the authors' artifacts. A good artifact offers a quick first run, states hardware and time requirements and maps each claim to a script; the sysartifacts guides show how. For ML, the NeurIPS paper checklist asks about claims, limitations, reproducibility details, access to data and code, statistical significance and compute.

Common misconceptions

• "The code is on GitHub, so it is reproducible." Without exact configuration, data version, environment and seeds, others get different numbers and cannot tell why.
• "Containers solve reproducibility." They pin software, not hardware, external services or data, and a mutable base-image tag drifts.
• "A seed makes GPU training deterministic." Some kernels and reductions stay nondeterministic unless configured, and hosted models can change behind the same name.
• "Reproduced means identical numbers." Evaluations accept agreement within a tolerance that supports the same conclusion; define that tolerance beforehand.
• "Provenance is for the final runs." Exploratory runs are where you will need to trace an odd result; log from the start.

How to approach it

Retrofit one past project: add a harness that writes the provenance above, regenerate one figure from raw logs with one command and write an artifact README following the sysartifacts packaging guide. In interviews, expect "how would someone else reproduce this?"; strong answers name the specific records kept and the known sources of irreproducibility.