Underfitting
Underfitting occurs when a model is too simple to capture the underlying pattern, performing poorly on both training and test data.
Definition
An underfit model has high training error and high test error — it hasn't learned the signal, not just the noise.
Causes: model too simple for the true relationship, features not informative, training stopped too early.
Intuition
Underfitting is the opposite problem from overfitting — you need a more powerful model or better features, not more regularization.
In the classical bias-variance tradeoff, increasing model flexibility can reduce bias while increasing variance; stronger regularization often does the reverse.
Worked example
Fitting a straight line to data that has a quadratic relationship — no matter how much data you add, the line will always miss the curve.
A sentiment classifier that just counts positive and negative words, missing context and negation.
The math
Irreducible error sets a floor: even the best possible model has some error due to noise in .
The "just noticeable difference" in model capacity: compare training error to a human-baseline or theoretical minimum.
In practice
Diagnose underfitting by comparing training error to the Bayes error (ideally zero for deterministic problems).
Fix underfitting by increasing model capacity, adding non-linear features, reducing regularization, or engineering better features.
Sources
More in Data science
Assembled from the ReLU.chat curated knowledge base. These explanations are concise on purpose; check the sources for anything important.