Dataset shift
Dataset shift occurs when the distribution encountered during use differs from the training distribution. It can affect inputs, labels, or their relationship.
Definition
Dataset shift occurs when the distribution encountered during use differs from the training distribution. It can affect inputs, labels, or their relationship.
Intuition
An accurate past model may fail when the world or the measurement process changes.
Worked example
A sensor replacement changes the range of measured values even though the physical process is similar.
The math
Covariate shift changes P(X); label shift changes P(Y); concept drift commonly refers to a change in P(Y|X). Real changes may combine them.
In practice
Monitor input quality and labeled performance where available. A drift detector signals change, not necessarily a need to retrain.
Sources
More in Data science
Assembled from the ReLU.chat curated knowledge base. These explanations are concise on purpose; check the sources for anything important.