Principal Component Analysis Explained: Finding Directions of Most Variance

Principal component analysis (PCA) rotates a dataset onto new axes, called principal components, ordered by how much variance each one captures. The first component points along the direction in which the data spreads the most, and each later component is perpendicular to the ones before it. In practice you center the data, compute its covariance matrix, and take the eigenvectors. Keeping only the first few components reduces the number of features while preserving most of the variation.

This guide explains the idea, works a small example by hand, and covers the choices that change the result, especially scaling.

The problem PCA solves

Real datasets often have many features that move together. Height and weight are correlated, and so are many sensor readings and survey answers. When features are correlated, the cloud of points stretches along a few directions and stays flat along others. Describing every point with all of the original coordinates repeats much of the same information.

PCA gives you a new coordinate system aligned with that stretch. Once the data is expressed in those coordinates, you can keep the directions with large spread and set aside the flat ones. That makes visualization easier, reduces storage and computation, and sometimes filters out noise.

The idea: directions of spread

A principal component is a unit vector. Projecting each data point onto it produces a score. The first principal component is the direction that makes those scores vary the most. Equivalently, it is the line that fits the data best in the sense of minimizing the average squared perpendicular distance from the points to the line. The Wikipedia article on principal component analysis gives this definition and notes that each later component is chosen to be orthogonal to the earlier ones.

A little history

Karl Pearson introduced the method in 1901, as an analogue of the principal axis theorem in mechanics. Harold Hotelling later developed it independently and gave it its name in the 1930s.

Computing principal components

1. Center each feature by subtracting its mean.
2. If the features use different units, standardize them so that each has unit variance.
3. Form the covariance matrix C of the centered data. For a centered data matrix X with n rows, C = XᵀX / (n − 1).
4. Compute the eigenvalues and eigenvectors of C.
5. Sort the eigenvectors by eigenvalue, largest first. The first eigenvector is the first principal component.
6. Project the data onto the top k eigenvectors to get the reduced representation.

Steps 3 and 4 use the eigen ideas from the companion guide on eigenvalues and eigenvectors. Because C is symmetric, its eigenvectors are perpendicular, and its eigenvalues are the variances along those directions.

A two-feature example

Suppose two features have been standardized, so each has variance 1, and their correlation is 0.8. The covariance matrix is then

C = | 1.0  0.8 |
    | 0.8  1.0 |

Its eigenvalues solve (1 − λ)² − 0.8² = 0, so 1 − λ = ±0.8, giving λ = 1.8 and λ = 0.2. Their sum is 2, which is the total variance of the two standardized features.

For λ = 1.8, the eigenvector is proportional to (1, 1), so the first principal component is (1, 1)/√2. For λ = 0.2, the second component is (1, −1)/√2.

The first component carries 1.8 / 2 = 90% of the total variance. A two-feature dataset can therefore be reduced to one number per row while keeping 90% of the variance. The second component holds the small leftover variation perpendicular to the first.

Scale matters

PCA reads variance literally. A feature measured in millimeters has numbers about a thousand times larger than the same length in meters, so it dominates the first component unless the features are on comparable scales. Standardizing each feature, by subtracting its mean and dividing by its standard deviation, gives every feature the same weight. For standardized data, the covariance matrix becomes the correlation matrix.

Whether to standardize is a modeling decision. If every feature shares one meaningful unit, raw variance may be exactly what you want. The guide on feature normalization for machine learning beginners covers standardization and related scaling choices.

Choosing how many components to keep

Compute the share of variance each component explains: its eigenvalue divided by the sum of all eigenvalues. Plot these shares, often as a scree plot, and look for the point where additional components add little. Many people keep enough components to reach a target share such as 90% or 95%. That threshold is a convention, not a law.

If PCA is a preprocessing step for a model, check the downstream result as well. Keep the number of components that performs best on validation data.

Computing PCA with SVD

The same components can be computed without forming the covariance matrix, by taking the singular value decomposition of the centered data matrix. Many libraries do this, because forming XᵀX squares the condition number of the problem, and the SVD avoids that loss of precision. The guide on rank, SVD, and conditioning explains the stability side of this choice.

Where PCA helps, and where it does not

PCA is useful for visualizing high-dimensional data in two or three dimensions, for removing redundant correlated features, and as a preprocessing step before clustering. Reducing dimensions before running the k-means clustering tool can make groups easier to see.

Its limits matter as much:

• PCA is linear. If the structure is curved, a straight-line projection can miss it. The Wikipedia article warns that a nonlinear pattern can lead PCA in the wrong direction, and nonlinear methods exist for that case.
• PCA is unsupervised. Its directions depend only on the features, not on the labels you want to predict. A high-variance direction may have nothing to do with the target.
• Outliers have a large influence on the covariance matrix and can pull the components toward them.
• The new axes are mixtures of the original features, which can make them harder to interpret.

Try it yourself

The Data Science assistant covers principal component analysis along with dimensionality reduction, feature scaling, and related evaluation topics. Start with the PCA topic page, then read the pages on dimensionality reduction and feature scaling. To ask follow-up questions, open the Data Science chat. Questions are processed in your browser, and How it works explains which steps run on your device and which downloads the site still makes.

Frequently asked questions