Linear algebra, probability and statistics for ML

Why it matters

ML interview questions in this area are less about proofs than about whether you can work with the objects behind a model: the shapes in a batched layer, the gradient of a loss, why a loss has the form it has, and whether a reported gain is larger than noise. Gaps here show up as shape bugs, losses chosen by habit, and over-interpreted evaluation numbers. For research-leaning roles, expect short derivations on a whiteboard.

Core ideas

Shapes first. With a batch X of shape (n, d) and weights W of shape (d, k), X @ W is (n, k), and a bias of shape (k,) broadcasts across rows. Writing the shape next to every line is the fastest way to catch bugs, and the gradient of a scalar loss always has the same shape as the parameter it is taken with respect to.

Gradients of common expressions.

• f(w) = a^T w has gradient a.
• f(w) = w^T A w has gradient (A + A^T) w, which is 2 A w only when A is symmetric.
• Least squares L(w) = ||X w - y||^2 has gradient 2 X^T (X w - y); setting it to zero gives the normal equations X^T X w = X^T y.
• Softmax cross-entropy with logits z and one-hot target y has gradient softmax(z) - y with respect to z. Derive this once; it explains why frameworks fuse the two into a single operation on logits.

Eigenvalues and SVD. A symmetric matrix has orthogonal eigenvectors, and its eigenvalues say how much it stretches each of those directions. Every matrix has an SVD A = U S V^T: rotate, scale by the singular values, rotate. Consequences worth stating fluently: keeping the top k singular values gives the best rank-k approximation (the basis of PCA and low-rank compression, and a useful intuition for low-rank adapters such as LoRA); PCA directions are the right singular vectors of the centered data matrix; and the condition number, the ratio of largest to smallest singular value, measures how much small errors can be amplified. Nearly collinear features make X^T X ill-conditioned, which destabilizes least squares and slows gradient descent.

Probability and likelihood. Expectation is linear, Var(aX + b) = a^2 Var(X), and the variance of a sum includes covariance terms. Maximum likelihood chooses parameters maximizing sum_i log p(y_i | x_i; theta). Gaussian output noise with fixed variance turns this into minimizing squared error; Bernoulli or categorical outputs turn it into (binary) cross-entropy. MAP adds a log-prior: a zero-mean Gaussian prior on the weights gives an L2 penalty, a Laplace prior gives L1. This is how you justify a loss instead of reciting it.

Statistics of evaluation. A test accuracy is a mean of n Bernoulli outcomes, so its standard error is about sqrt(p (1 - p) / n). At p = 0.9 and n = 1000 that is roughly 0.0095, so a 95% interval spans about plus or minus 1.9 points: a half-point difference on such a set is not evidence. When two models are scored on the same examples, analyze per-example differences (a paired test or a bootstrap over examples), which removes the variance shared by both models. A p-value is the probability of data at least this extreme if the null hypothesis were true; it is not the probability that the null is true, and it says nothing about effect size.

Common misconceptions

• "The gradient of x^T A x is 2Ax." Only for symmetric A; in general it is (A + A^T) x.
• "Cross-entropy and MSE are arbitrary choices." Each is a negative log-likelihood under an assumed output distribution (Gaussian or categorical); choosing one is an assumption about the outputs.
• "Eigenvalues and singular values are the same thing." They coincide for symmetric positive semidefinite matrices and match in absolute value for symmetric ones; in general they differ, and non-square matrices have singular values but no eigenvalues.
• "p = 0.03 means a 97% chance the new model is better." The p-value is computed assuming no difference; it is not a posterior probability.
• "More test data is the only fix for noisy comparisons." The standard error shrinks like 1/sqrt(n), so halving it needs four times the data; pairing and variance reduction are often cheaper.
• "Overlapping confidence intervals mean no significant difference." Overlap of separate intervals is a conservative check; a paired analysis can detect a real difference even when the marginal intervals overlap.

How to approach it

Refresh with the CS229 linear algebra and probability reviews, then derive by hand: least squares, logistic regression, softmax cross-entropy. Verify each with central finite differences in float64. Revisit SVD with a small experiment, compressing a matrix at several ranks and comparing the error with the discarded singular values. For statistics, compute a bootstrap interval for a metric you care about. In interviews, state shapes and assumptions aloud; graduate-level interviews emphasize derivations, senior-level ones emphasize whether the evidence supports the decision.