Least squares
Least squares chooses parameters that minimize the sum of squared residuals when exact agreement may be impossible.
Definition
Least squares chooses parameters that minimize the sum of squared residuals when exact agreement may be impossible.
Intuition
It finds the representable prediction closest to the observations.
Worked example
Fitting one constant to observations 1, 2, 6 gives 3, their mean; the residuals sum to zero.
The math
Minimize . At a solution, ; uniqueness requires full column rank.
In machine learning
QR or SVD is often more numerically robust than forming normal equations, which square the condition number.
Go deeper
- GuideLeast Squares Without Computing an Inverse
- InteractiveGradient Descent Lab
Sources
More in Linear algebra
Assembled from the ReLU.chat curated knowledge base. These explanations are concise on purpose; check the sources for anything important.