Dimensionality Reduction
Dimensionality reduction transforms high-dimensional data into a lower-dimensional representation while preserving key structure.
Definition
PCA projects onto orthogonal axes of maximum variance. t-SNE and UMAP are non-linear methods that emphasize preserving local neighborhood structure for visualization.
Autoencoders learn a compressed latent representation by training a neural network to reconstruct its input.
Intuition
The "curse of dimensionality" says that distance-based methods degrade as dimensions grow — fewer dimensions with the same information is better.
Non-linear methods (t-SNE, UMAP) can reveal clusters and manifolds that PCA misses, but are not reversible (no inverse mapping).
Worked example
Compressing 784-pixel MNIST images to 50 dimensions via PCA retains most variance; a classifier trained on 50-D data is faster and more robust.
Visualizing a 100-feature dataset in 2D using UMAP reveals natural clusters.
The math
PCA closed-form solution: eigenvectors of sorted by eigenvalue. t-SNE uses KL divergence between neighborhood distributions in high and low dimensions.
Intrinsic dimensionality: the number of dimensions actually needed to describe the data — estimable by the "elbow" in the explained variance plot.
In practice
Used for visualization (2-3D), noise reduction, compression, and as a preprocessing step to reduce overfitting in supervised learning.
For supervised dimensionality reduction, Linear Discriminant Analysis (LDA) maximizes class separability rather than variance.
More in Data science
Assembled from the ReLU.chat curated knowledge base. These explanations are concise on purpose; check the sources for anything important.