Linear regression: the first model, a straight line

Linear regression is usually the first model anyone learns, and sections 3.1 to 3.3 of An Introduction to Statistical Learning (ISLP) cover it well. It’s worth learning properly, because it’s simple, fast, easy to explain to a business user, and the starting point for many other models.

This post continues the regression vs classification example: predicting how volatile the market will be next month.

The idea

You have some months. For each one you know how volatile it was (the standard deviation of its daily returns, annualised), and how volatile the next month turned out to be. Plot them, and they roughly form a line: a wild month tends to be followed by a wild month. Linear regression finds the one straight line that fits the points best, so you can read off a forecast for any level of current volatility.

A straight line has two numbers. In the notation of An Introduction to Statistical Learning (ISLP), equations 3.1 and 3.2, with hats marking numbers estimated from data:

β̂ and ŷ, what the model estimatesy, the actual value

\key{\hat{y}} = \key{\hat{\beta}_0} + \key{\hat{\beta}_1} x

Symbol	Meaning	In our example
$x$	the input (feature)	this month’s volatility, % a year
$\key{\hat{y}}$	the prediction (“y-hat”)	next month’s volatility, % a year
$\key{\hat{\beta}_1}$ (“beta-one-hat”)	the slope: how much $\key{\hat{y}}$ goes up when $x$ goes up by 1	points of next-month volatility per point of this month’s
$\key{\hat{\beta}_0}$	the intercept: the value of $\key{\hat{y}}$ when $x = 0$	the line’s starting level

What “best” means: least squares

For each month, the line makes a prediction $\key{\hat{y}_i}$ . The real value is $\good{y_i}$ . The difference $\good{y_i} - \key{\hat{y}_i}$ is the residual, the miss.

Linear regression picks the line that makes the residual sum of squares (RSS) as small as possible (ISLP equation 3.3):

RSS, the error left over

\risk{\text{RSS}} = \sum_{i=1}^{n} \left( \good{y_i} - \key{\hat{y}_i} \right)^2

Why square them? Three reasons. A miss above the line and a miss below don’t cancel out. Big misses count much more than small ones. And squaring gives a neat formula for the answer, with no searching needed. This method is called ordinary least squares (OLS).

For one input, the best slope and intercept are (ISLP equation 3.4):

\key{\hat{\beta}_1} = \frac{\sum_i (x_i - \bar{x})(\good{y_i} - \bar{y})}{\sum_i (x_i - \bar{x})^2}

\key{\hat{\beta}_0} = \bar{y} - \key{\hat{\beta}_1} \bar{x}

where $\bar{x}$ and $\bar{y}$ (read “x-bar”, “y-bar”) are the averages.

Plain meaning of the slope formula: the top measures how $x$ and $y$ move together. When a month is wilder than usual, is the next one also wilder than usual? The bottom measures how much $x$ moves on its own. Their ratio is “how much $y$ moves per unit of $x$ ”. (The top part, divided by $n-1$ , is the covariance, which gets its own post in this series.)

Worked example: five months by hand

Five SPY months at two-month gaps through the 2020 recovery, volatility rounded to whole percentage points so you can follow the arithmetic:

Month starting	This month $x$	Next month $y$	$x - \bar{x}$	$y - \bar{y}$	product	$(x - \bar{x})^2$
Jun 2020	20	30	2	13.4	26.8	4
Aug 2020	13	8	−5	−8.6	43.0	25
Oct 2020	24	21	6	4.4	26.4	36
Dec 2020	16	8	−2	−8.6	17.2	4
Feb 2021	17	16	−1	−0.6	0.6	1
Average / Sum	$\bar{x} = 18$	$\bar{y} = 16.6$			114.0	70

\begin{aligned} \key{\hat{\beta}_1} &= \frac{114.0}{70} \\[10pt] &= 1.63 \end{aligned}

\begin{aligned} \key{\hat{\beta}_0} &= 16.6 - 1.63 \times 18 \\ &= -12.7 \end{aligned}

\key{\widehat{\text{next}}} = -12.7 + 1.63 \times \text{this month}

Reading it: on these five months, each extra point of volatility this month came with about $\key{1.6}$ extra points next month. A month at 20%: $-12.7 + 1.63 \times 20 = 19.9\%$ expected next.

Be careful with the intercept. It says that a month with zero volatility is followed by one at −12.7%, which makes no sense (volatility can’t be negative). We never saw a month anywhere near zero. A line is only trustworthy inside the range of data it learned from. Using it far outside that range is called extrapolation, and it is the usual way a linear model goes wrong in practice.

The dotted lines on the left are the residuals. The fitted line is the one where the squares of those dotted lines add up to the smallest possible total, here $\risk{161.5}$ .

How good is the fit? R²

$R^2$ (“R squared”, ISLP equation 3.17) answers: what share of the ups and downs in $y$ does the line explain?

RSS and TSS, the error left over and the total spread

\begin{aligned} R^2 &= 1 - \frac{\risk{\text{RSS}}}{\risk{\text{TSS}}} \\[10pt] &= 1 - \frac{\sum_i (\good{y_i} - \key{\hat{y}_i})^2}{\sum_i (\good{y_i} - \bar{y})^2} \end{aligned}

TSS (total sum of squares) is how much $y$ varies around its own average: the error you’d make if you ignored $x$ and always guessed the average.
RSS (residual sum of squares) is the error left over after using the line.

For the five months, with RSS $= 161.5$ :

\begin{aligned} \risk{\text{TSS}} &= 13.4^2 + 8.6^2 + 4.4^2 + 8.6^2 + 0.6^2 = 347.2 \\[10pt] R^2 &= 1 - \frac{161.5}{347.2} = 0.535 \end{aligned}

The line explains $\key{53\%}$ of the month-to-month variation in volatility. Real market data is noisy; nothing here will score in the nineties. $R^2 = 0$ means no better than guessing the average. $R^2 = 1$ means perfect.

More than one input

A month is more than its volatility. Markets that have just fallen tend to stay nervous, so add last month’s return as a second input:

\key{\hat{y}} = \key{\hat{\beta}_0} + \key{\hat{\beta}_1} x_1 + \key{\hat{\beta}_2} x_2

Now the “line” is a flat surface in three dimensions, but the idea is the same: find the numbers that make the squared misses smallest.

Trained on the 3,685 ticker-months of 2019 to 2021 (all five tickers), then tested on 2022 to 2023:

Item	One input	Two inputs
Intercept $\key{\hat{\beta}_0}$	9.74	11.03
Per point of this month’s volatility $\key{\hat{\beta}_1}$	0.604	0.577
Per point of last month’s return $\key{\hat{\beta}_2}$		−0.315
$R^2$ on the training months	0.367	0.386
$R^2$ on the test months	$\key{0.458}$	$\key{0.414}$

Reading $\key{\hat{\beta}_2}$ : a month that fell 10% instead of staying flat means $-0.315 \times -10 = 3.2$ more points of next-month volatility, with this month’s volatility held the same. That last part matters. In multiple regression, each number is the effect of one input while the others stay fixed. The sign is right (falls make markets nervous), and the size is sensible.

Now look at the last row. The second input improved the fit on the training months, as adding an input always does, and made the fit on the test months slightly worse: $R^2$ fell from 0.458 to 0.414. The return carries a little information, but the model also learned some of 2019 to 2021’s particular quirks through it, and those didn’t repeat. More inputs are not automatically better; only the test score can tell you. That tension gets a whole post, bias and variance.

(Why is the test $R^2$ higher than the training one for the one-input model? 2020’s crash made the training years unusually hard to fit; 2022 and 2023 were tamer. $R^2$ depends on the data as much as on the model.)

The matrix form

With many inputs, the formula is written with matrices. Put the inputs in a matrix $X$ (one row per day, plus a column of 1s for the intercept) and the answers in a vector $y$ . The best coefficients are:

\key{\hat{\beta}} = (X^\top X)^{-1} X^\top y

This comes from the normal equations $X^\top X \key{\hat{\beta}} = X^\top y$ (Murphy, Probabilistic Machine Learning, chapter 11). ISLP doesn’t print the matrix form, because it needs matrix algebra. In practice, libraries solve it with more numerically stable methods than inverting the matrix, but the answer is the same. On the five months it gives exactly $(-12.7,\ 1.63)$ .

The code

import numpy as np
from sklearn.linear_model import LinearRegression

x = np.array([20, 13, 24, 16, 17])         # this month's volatility, % a year
y = np.array([30, 8, 21, 8, 16])           # next month's

# By hand
b1 = ((x - x.mean()) * (y - y.mean())).sum() / ((x - x.mean()) ** 2).sum()
b0 = y.mean() - b1 * x.mean()
print(b0, b1)                               # -12.7  1.63

# With scikit-learn: same answer
model = LinearRegression().fit(x.reshape(-1, 1), y)
print(model.intercept_, model.coef_)        # -12.7  [1.63]
print(model.predict([[20]]))                # [19.9]

When linear regression goes wrong

The relationship isn’t a straight line. If volatility only matters above some level, a line will miss it. (Decision trees, the next post, handle this.)
Extrapolation. Never trust the line far outside the data, like the “−12.7% next month after a month with no volatility”.
Outliers. Because misses are squared, one extreme month (March 2020) can pull the line a long way. The five-month example avoided it; the three-year fit had to live with it.
Inputs that move together. If two inputs are strongly linked (5-day and 20-day volatility, say), the model can’t tell their effects apart, and the individual numbers become unstable, even when the predictions are fine.
Correlation is not causation. The slope says this month’s and next month’s volatility move together. It doesn’t say one causes the other; both may be reacting to the same news.

Summary

Idea	Formula	Remember it as
The line	$\key{\hat{y}} = \key{\hat{\beta}_0} + \key{\hat{\beta}_1} x$	Start level plus slope times input
Best fit	minimise $\sum (\good{y_i} - \key{\hat{y}_i})^2$	Least squares
Slope	$\key{\hat{\beta}_1} = \sum (x - \bar{x})(y - \bar{y}) / \sum (x - \bar{x})^2$	How they move together ÷ how $x$ moves alone
Intercept	$\key{\hat{\beta}_0} = \bar{y} - \key{\hat{\beta}_1} \bar{x}$	The line passes through the averages
Fit quality	$R^2 = 1 - \risk{\text{RSS}}/\risk{\text{TSS}}$	Share of the variation explained
Many inputs	$\key{\hat{\beta}} = (X^\top X)^{-1} X^\top y$	The normal equation

Decision trees: a model that asks a series of yes/no questions instead of drawing a line, and can follow patterns a straight line can’t.

References

James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), sections 3.1 to 3.3: the model (equations 3.1, 3.2), RSS (3.3), least-squares estimates (3.4), $R^2$ (3.17)
Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), chapter 11: the normal equations
scikit-learn: ordinary least squares