Linear Regression

Linear regression predicts a continuous target as a weighted sum of features, fit by ordinary least squares (OLS).

Ask the Data science assistant1 min read · Updated September 9, 2026

Definition

$\hat{y} = w^Tx + b$ where $w$ minimizes $\sum_i (y_i - w^Tx_i - b)^2$ , the residual sum of squares (RSS).

Closed-form solution: $w = (X^TX)^{-1}X^Ty$ (when $X^TX$ is invertible).

Intuition

Each coefficient $w_j$ tells you the expected change in $y$ for a one-unit increase in $x_j$ , holding all else constant.

OLS is the best linear unbiased estimator (BLUE) under Gauss-Markov assumptions: linearity, exogeneity, homoscedasticity, no perfect multicollinearity.

Worked example

Predicting weight from height: $\hat{y} = -100 + 0.9 \times \text{height(cm)}$ , so each extra cm adds about 0.9 kg.

If $w_2 = 3.5$ , then feature $x_2$ has a positive association with $y$ after controlling for other features.

The math

Assumptions: $y = Xw + \epsilon$ , with $E[\epsilon|X] = 0$ , $\text{Var}(\epsilon|X) = \sigma^2 I$ , and $\epsilon \sim N(0, \sigma^2)$ for inference.

Hypothesis test on $w_j$ : $H_0: w_j = 0$ (feature has no effect); p-value from $t$ -statistic $t = w_j / \text{SE}(w_j)$ .

In practice

Baseline regression model; interpretable coefficients make it the go-to for explanatory modeling.

Check residual plots for non-linearity, heteroscedasticity, and outliers — violations indicate model misspecification.

Go deeper

GuideLeast Squares Without Computing an Inverse
GuideDot Product and Cosine Similarity, Step by Step
GuideRank, SVD, and Conditioning: Three Different Questions
InteractiveGradient Descent Lab

More in Data science

Assembled from the ReLU.chat curated knowledge base. These explanations are concise on purpose; check the sources for anything important.