Metrics, calibration, thresholds and uncertainty

Why it matters

A metric is a claim about what matters. Fraud detection, search ranking and medical triage need different metrics, and the same model can look excellent or useless depending on the choice. Interviews test whether you can pick the metric from the decision, explain a surprising number, and turn scores into actions with thresholds tied to costs.

Core ideas

Classification. From the confusion matrix: precision TP / (TP + FP) asks how many flagged items are real; recall TP / (TP + FN) asks how many real items were caught. F1 is their harmonic mean, and F-beta weights recall beta times as heavily as precision. Accuracy is dominated by the majority class: with 1% positives, always predicting "negative" scores 99%.

ROC-AUC versus PR-AUC. The ROC curve plots true-positive rate against false-positive rate, and its AUC equals the probability that a random positive is scored above a random negative. Because the false-positive rate divides by the number of negatives, which is huge under imbalance, a model can raise thousands of false alarms while its FPR stays tiny and ROC-AUC looks excellent. The precision-recall curve exposes those false alarms, and its baseline is the positive rate rather than 0.5. When positives are rare and they are what you act on, report average precision (PR-AUC) and precision at the operating point. ROC-AUC remains useful for comparing rankers independently of prevalence. Neither measures calibration.

Regression. RMSE squares errors, so large misses dominate it, and it is minimized by predicting the mean; MAE is linear in the error and minimized by the median. Choose by the cost of big errors. MAPE explodes near zero targets and treats over- and under-prediction asymmetrically.

Ranking. For each query's ranked results:

• recall@k: the fraction of relevant items that appear in the top k. In RAG it is the first retrieval metric to check, because the generator only sees the top k.
• MRR: the mean over queries of 1 / rank of the first relevant item; good when one correct result suffices.
• NDCG@k: DCG = sum over positions i of rel_i / log2(i + 1), divided by the DCG of the ideal ordering; it handles graded relevance and rewards placing the best items first.

Calibration. A classifier is calibrated if, among predictions of 0.8, about 80% are positive. A reliability diagram bins predictions and plots observed frequency against mean confidence; expected calibration error (ECE) summarizes the gaps but depends on the binning and can hide problems. Log loss and Brier score are proper scoring rules: they reward both ranking quality and calibration. Modern deep networks are often overconfident. Temperature scaling divides the logits by one scalar T fitted to minimize log loss on held-out data; it changes confidence without changing the argmax, so accuracy is untouched. Platt scaling (a logistic fit on scores) and isotonic regression (monotone, needs more data) are alternatives. Calibration holds only for the distribution it was fitted on; a change in class prior breaks it.

Thresholds from costs. With calibrated probability p, a false-positive cost C_FP and a false-negative cost C_FN, flag when p * C_FN > (1 - p) * C_FP, that is when p > C_FP / (C_FP + C_FN). If a false alarm costs one unit and a miss costs 50, the threshold is about 0.02, not 0.5. Without calibrated probabilities, choose the threshold on validation data to meet a constraint, such as precision of at least 90% or a review team's capacity per day.

Uncertainty. Report metrics with intervals (bootstrap over examples, or over queries for ranking). For individual predictions, let the system abstain or route to human review when confidence is low; ensembles and conformal prediction give more principled uncertainty.

Common misconceptions

• "High ROC-AUC means the model is usable." Under imbalance, precision at the recall you need can still be poor; check the PR curve at the operating point.
• "A score of 0.9 means a 90% chance." Only if the model is calibrated on this distribution.
• "Temperature scaling improves accuracy." It cannot change the argmax; it fixes confidence only.
• "F1 is the best default metric." It ignores true negatives and fixes the precision-recall trade at equal weight; real costs rarely agree.
• "Calibrate on the training set." Fit calibration on held-out data, or it learns the overconfident training behavior.
• "NDCG and recall@k measure the same thing." Recall@k ignores order within the top k; NDCG rewards ordering and graded relevance.

How to approach it

For any scenario ask: what action follows a prediction, what do errors cost, how rare is the positive class, and are probabilities consumed downstream? Derive the metric from the answers. Practice ranking metrics by hand, plot a reliability diagram for a real classifier, and fit a temperature. Senior-level discussions add prevalence shift and metric uncertainty.