Overfitting
Overfitting occurs when a model learns the training data too well — including its noise — and generalizes poorly.
Definition
A overfit model has low training error but high test error; it has memorized patterns that don't hold in the real world.
Symptoms: complexity too high (tree too deep, neural net too wide), too many features, insufficient regularization.
Intuition
Every dataset has genuine signal and random noise; the model's job is to capture signal without fitting noise.
Double descent: very large models can again decrease test error after an initial overfitting peak — modern deep learning exploits this.
Worked example
A degree-9 polynomial fit to 10 data points goes through every point (zero training error) but oscillates wildly between them.
A decision tree trained to pure depth on a small dataset memorizes every training example including outliers.
The math
The bias-variance decomposition: .
Overfitting corresponds to high variance — small changes in training data cause large changes in the learned model.
In practice
Combat overfitting with: more training data, simpler models, regularization, dropout (neural nets), early stopping, or cross-validation.
Always monitor validation error during training — if it starts rising while training error falls, you're overfitting.
Go deeper
- InteractiveDecision Tree Explorer
More in Data science
Assembled from the ReLU.chat curated knowledge base. These explanations are concise on purpose; check the sources for anything important.