Least squares fits a model when the observations do not lie exactly in the model’s representable space. The goal is to minimize the sum of squared residuals, not to force an exact solution where none exists.

Consider a line y = a + bx through observations (0, 1), (1, 2), and (2, 2). Two parameters cannot generally satisfy three inconsistent equations exactly.

Write the system

The design matrix has rows (1, 0), (1, 1), and (1, 2). The parameter vector is (a, b), and the observation vector is (1, 2, 2). Predictions are the matrix-vector product Aθ.

The least-squares objective is the squared Euclidean norm of Aθ − y. Differentiating gives the normal equations: AᵀAθ = Aᵀy. Here AᵀA has rows (3, 3), (3, 5), and Aᵀy is (5, 6).

Solve the small example

The equations are 3a + 3b = 5 and 3a + 5b = 6. Subtracting gives 2b = 1, so b = 1/2. Substitution gives a = 7/6.

The predictions are 7/6, 5/3, and 13/6. Observed minus predicted residuals are −1/6, 1/3, and −1/6. Their squared sum is 1/6.

The residuals sum to zero because the model includes an intercept. Their dot product with the x column is also zero. These are useful checks of the normal equations, not arbitrary coincidences.

Why avoid the inverse in code?

The expression involving (AᵀA) inverse is a convenient algebraic description when full column rank holds. It is usually not the best numerical recipe. Forming AᵀA squares the condition number, and explicitly computing an inverse adds work that the desired solution does not require.

QR solves least squares through an orthogonal factorization. SVD also reveals small singular values and supports rank-deficient cases. A library’s least-squares solver is generally a better starting point than manually multiplying by an inverse.

Check the model as well as the arithmetic

A small residual does not prove that the relationship is causal, that a line extrapolates safely, or that training and evaluation data were separated correctly. Numerical correctness and statistical validity are different checks.

For a concrete coding exercise, calculate the example with a library solver, then verify the residual and the normal-equation condition. Perturb one observation and see how the coefficients change.

See MIT’s linear algebra course, our data leakage guide, and the linear algebra chatbot.