Probability calibration
Calibration asks whether predicted probabilities agree with observed frequencies. A model can rank cases well while assigning misleading probability values.
Definition
Calibration asks whether predicted probabilities agree with observed frequencies. A model can rank cases well while assigning misleading probability values.
Intuition
Among cases assigned a probability near 0.7, roughly 70% should be positive in a well-calibrated, representative sample.
Worked example
A classifier predicts 0.9 for 100 cases but only 60 are positive. Those predictions are overconfident even if the ranking is useful.
The math
For binary outcomes, Brier score is the mean of (p−y)². It reflects calibration and other predictive properties, not calibration alone.
In practice
Fit a calibrator without leaking test labels, then assess reliability and task-specific decision costs.
Sources
More in Data science
Assembled from the ReLU.chat curated knowledge base. These explanations are concise on purpose; check the sources for anything important.