Repeat one benchmark ten times and report a paired confidence interval

Measure run-to-run variability, choose the experimental unit, interleave two variants and report the interval of their paired difference.

Not attemptedSelf-directed (reporting rules from Hoefler and Belli)Access: free

Open resource

The exercise happens on Self-directed (reporting rules from Hoefler and Belli). Canonical link: https://htor.inf.ethz.ch/publications/img/hoefler-scientific-benchmarking.pdf. Link check: link responded (5 Oct 2026). Opened from this app 0 times; opening is not counted as an attempt.

Supports: Credible performance and ML comparisons; questions about variance and seeds.

Record an attempt

OutcomeAttempted, not solvedSolved with help (hints, editorial or discussion)Solved independentlyReviewed an earlier solution

Date

Minutes (optional)

Language

Hints used

Confidence afterwards

Main mistake

Learning objective this practices (optional)

Notes (approach, bug, what to remember)

Re-attempt on (optional)

A date is suggested from the outcome.

Attempt history

No attempts recorded yet.