Evaluating ML and LLM systems

Why it matters

Without evaluation, every change to a prompt, model, retriever or tool is a guess, and regressions reach users first. Applied AI and ML interviews ask how you would know a system works, how you would compare two versions, and why anyone should trust a model-based grader. Clear answers show product sense and scientific discipline at the same time.

Core ideas

Start from real cases. Build the dataset from logged or realistic inputs, support tickets and observed failures, covering the slices that matter (languages, customer types, edge cases), not from a handful of demo prompts. Practitioner guidance suggests that twenty to fifty well-chosen tasks are enough to start while changes still have large effects; grow the set as the system matures and differences shrink. Keep a frozen held-out portion for decisions you do not want to overfit.

Define success precisely. For each task, write down what passing means in terms two experts would agree on. Prefer checking outcomes (the record exists, the tests pass, the answer contains the right figure) over checking the exact path the system took. Use separate criteria or partial credit when a task has several parts, such as correct facts, correct format and the correct tool action.

Error analysis first. Read a sample of outputs, write a short note on each failure, group the notes into categories and count them. This shows which failures are frequent and which are severe, and often reveals that a "model problem" is really a retrieval, data or prompt problem. Repeat after every significant change.

Graders.

• Code-based: exact match, regular expressions, schema validation, unit tests, state checks. Fast, cheap and reproducible, but brittle to valid variation.
• Human: domain experts or trained annotators with a rubric. The reference standard, but slow and costly; measure inter-annotator agreement to learn how well-defined the task is.
• Model-based (LLM-as-judge): scalable for open-ended criteria such as faithfulness or tone. Known biases include position bias (favoring one slot in a pairwise comparison), verbosity bias (favoring longer answers) and self-preference; judgments can also vary between runs. Mitigate by swapping answer order, using specific rubrics with one criterion per call, asking for evidence before the verdict, allowing an "unknown" verdict, and above all calibrating against human labels on a sample. Report agreement, for example Cohen's kappa, and re-check whenever the judge model or prompt changes.

Nondeterminism. Sampling makes outputs vary between runs on the same input, and serving infrastructure can add variation even at temperature zero. Run several trials per task and report per-task success rates. Two summaries help: pass@k, the chance that at least one of k trials succeeds (relevant when a person or system can choose among attempts), and pass^k, the chance that all k succeed (relevant when users expect consistent behavior). Report means with confidence intervals, compare versions with paired analysis on the same tasks, and remember that small evaluation sets cannot detect small improvements.

Regression suites in CI. Separate capability evaluations (hard tasks, low pass rates, for measuring progress) from regression evaluations (tasks the system should always pass). Run the regression suite on every change to prompts, models, retrieval or tools; fail the build on drops beyond measured noise, track cost and latency too, and review output diffs, not only aggregate scores. Promote capability tasks into the regression suite once they are reliably solved.

Offline is not online. Offline evaluation approximates real use; production monitoring, user feedback and online experiments measure it. Close the loop by turning production failures into new evaluation tasks.

Common misconceptions

• "A public benchmark score predicts my system's quality." Benchmarks measure other tasks and may be contaminated; build an evaluation for your use case.
• "LLM judges are objective." They have systematic biases and must be calibrated against human labels for each criterion.
• "Temperature zero makes evaluation deterministic." It is not guaranteed; run repeated trials and report variance.
• "An average score is enough." Averages hide slice failures; report per-slice results and read transcripts.
• "Grading the exact sequence of steps is more rigorous." It penalizes valid alternative solutions; grade outcomes and constrain only the steps that matter, such as forbidden actions.
• "The score went up, so the system improved." Check the interval, the slices, and whether the grader or dataset changed.

How to approach it

Treat evaluation as a product: version the dataset, graders and prompts, and keep transcripts. Practice on a small system of your own with 30 tasks, written criteria, error analysis, one calibrated judge and a script that runs on every change. In interviews, structure answers as data, criteria, graders, statistics and process.