Baselines, bias-variance, regularization and optimization

Why it matters

Questions such as "your model overfits, what do you do?" or "why does your deep model lose to logistic regression?" test whether you reason from evidence or from habit. Strong answers start from a baseline, read the gap between training and validation error, change one thing, and predict the effect before running it.

Core ideas

Baselines first. Predict the majority class or the mean, then a simple heuristic, then a linear model or gradient-boosted trees on the same features. Baselines calibrate what "good" means, expose pipeline bugs (a deep model that cannot beat the mean is broken, not unlucky) and often ship: on tabular data a well-tuned linear or tree model is a serious competitor. For LLM features the baseline is often a carefully written prompt, or a non-ML rule.

Bias and variance. For squared loss, expected test error over random training sets decomposes into bias^2 + variance + irreducible noise. Operationally, high training error means underfitting (high bias): add capacity or features, train longer, regularize less. Low training error with a large validation gap means overfitting (high variance): add data, regularize, augment, simplify. Learning curves, error against training-set size, show whether more data will help: if validation error is still falling, it will. Large neural networks complicate the classical U-shaped picture (double descent, implicit regularization from SGD), but the train-validation gap remains the operational signal.

Regularization as capacity control.

• L2 (lambda * ||w||^2) shrinks all weights smoothly; it is MAP with a Gaussian prior and stabilizes collinear features.
• L1 (lambda * ||w||_1) drives some weights exactly to zero, giving sparse models; among correlated features it tends to keep one somewhat arbitrarily.
• Weight decay shrinks weights by a constant factor each step. It equals L2 for plain SGD but not for adaptive optimizers such as Adam, which is why AdamW decouples it.
• Early stopping limits how far parameters move from initialization; choose the stopping point on validation data.
• Dropout zeroes random activations during training to discourage co-adapted features; it is switched off at inference, and inverted dropout rescales during training so nothing changes at test time.
• Data augmentation encodes invariances you believe in (crops, flips, paraphrases). Augmentations that change the label, or copies that land in both splits, hurt.

Objectives. The loss encodes what errors cost. Squared error targets the conditional mean and punishes large misses heavily; absolute error targets the median and is robust to outliers; Huber blends the two. Cross-entropy trains probabilities. Ranking tasks use pairwise or listwise losses. Class weights or resampling change the effective class prior, which shifts predicted probabilities and therefore calibration.

Optimization. Gradient descent updates w <- w - eta * grad L(w). Too large a learning rate oscillates or diverges; too small stalls. On ill-conditioned problems the stable step size is set by the steepest direction, so progress along flat directions is slow; feature standardization, normalization layers, momentum and adaptive methods all help. Mini-batch gradients are noisy but unbiased estimates of the full gradient. Linear and logistic regression are convex, so every local optimum is global; neural network losses are not, but in practice learning rate, conditioning and initialization matter more than bad local minima.

Common misconceptions

• "Start with the most powerful model." Without a baseline you cannot tell whether it helps or whether the pipeline is broken.
• "Regularization always improves validation scores." It trades variance for bias; if the model already underfits, regularization makes it worse.
• "L2 regularization and weight decay are the same." Only under plain SGD; under Adam they differ, hence AdamW.
• "More data fixes everything." It reduces variance, not bias, label noise or a mismatch between training and deployment data.
• "Dropout stays on at inference." Standard practice turns it off; keeping it on (Monte Carlo dropout) is a deliberate technique for uncertainty estimates.
• "Training fails because of bad local minima." Usually the learning rate, conditioning, initialization or a bug is responsible.

How to approach it

Rehearse a fixed diagnostic script: baseline, training error, validation gap, learning curve, then one targeted change. Derive the bias-variance decomposition once by hand. Fit L1 and L2 logistic regression across strengths and plot coefficient paths. Train a small network with and without dropout and augmentation and compare curves, predicting them first. Graduate-level interviews expect the vocabulary and the diagnosis; mid and senior levels expect remedies chosen under data and compute constraints, plus what you would measure.