Linear regression: the first model, a straight line

Linear regression is usually the first model anyone learns, and sections 3.1 to 3.3 of An Introduction to Statistical Learning (ISLP) cover it well. It’s worth learning properly, because it’s simple, fast, easy to explain to a business user, and the starting point for many other models.

This post continues the regression vs classification example: predicting how volatile the market will be next month.

The idea

You have some months. For each one you know how volatile it was (the standard deviation of its daily returns, annualised), and how volatile the next month turned out to be. Plot them, and they roughly form a line: a wild month tends to be followed by a wild month. Linear regression finds the one straight line that fits the points best, so you can read off a forecast for any level of current volatility.

A straight line has two numbers. In the notation of An Introduction to Statistical Learning (ISLP), equations 3.1 and 3.2, with hats marking numbers estimated from data; Table 1 reads the symbols:

β̂ and ŷ, what the model estimatesy, the actual value

$$\key{\hat{y}} = \key{\hat{\beta}_0} + \key{\hat{\beta}_1} x$$

What “best” means: least squares

For each month, the line makes a prediction $\key{\hat{y}_i}$. The real value is $\good{y_i}$. The difference $\good{y_i} - \key{\hat{y}_i}$ is the residual, the miss.

Linear regression picks the line that makes the residual sum of squares (RSS) as small as possible (ISLP equation 3.3):

RSS, the error left over

$$\risk{\text{RSS}} = \sum_{i=1}^{n} \left( \good{y_i} - \key{\hat{y}_i} \right)^2$$

Why square them? Three reasons. A miss above the line and a miss below don’t cancel out. Big misses count much more than small ones. And squaring gives a neat formula for the answer, with no searching needed. This method is called ordinary least squares (OLS).

For one input, the best slope and intercept are (ISLP equation 3.4):

$$\key{\hat{\beta}_1} = \frac{\sum_i (x_i - \bar{x})(\good{y_i} - \bar{y})}{\sum_i (x_i - \bar{x})^2}$$ $$\key{\hat{\beta}_0} = \bar{y} - \key{\hat{\beta}_1} \bar{x}$$

where $\bar{x}$ and $\bar{y}$ (read “x-bar”, “y-bar”) are the averages.

Plain meaning of the slope formula: the top measures how $x$ and $y$ move together. When a month is wilder than usual, is the next one also wilder than usual? The bottom measures how much $x$ moves on its own. Their ratio is “how much $y$ moves per unit of $x$”. (The top part, divided by $n-1$, is the covariance, which gets its own post, covariance and correlation.)

Worked example: five months by hand

Five SPY months at two-month gaps through the 2020 recovery, volatility rounded to whole percentage points so you can follow the arithmetic (Table 2):

$$\key{\hat{\beta}_1} = \frac{114.0}{70}$$ $$\key{\hat{\beta}_1} = 1.63$$ $$\key{\hat{\beta}_0} = 16.6 - 1.63 \times 18$$ $$\key{\hat{\beta}_0} = -12.7$$ $$\key{\widehat{\text{next}}} = -12.7 + 1.63 \times \text{this month}$$

Reading it: on these five months, each extra point of volatility this month came with about $\key{1.6}$ extra points next month. A month at 20%: $-12.7 + 1.63 \times 20 = 19.9\%$ expected next.

Be careful with the intercept. It says that a month with zero volatility is followed by one at −12.7%, which makes no sense (volatility can’t be negative). We never saw a month anywhere near zero. A line is only trustworthy inside the range of data it learned from. Using it far outside that range is called extrapolation, and it is the usual way a linear model goes wrong in practice.

The dotted lines in the top panel of Figure 1 are the residuals. The fitted line is the one where the squares of those dotted lines add up to the smallest possible total, here $\risk{161.5}$.

How good is the fit? $R^2$

$R^2$ (“R squared”, ISLP equation 3.17) answers: what share of the ups and downs in $y$ does the line explain?

RSS and TSS, the error left over and the total spread

$$R^2 = 1 - \frac{\risk{\text{RSS}}}{\risk{\text{TSS}}}$$ $$R^2 = 1 - \frac{\sum_i (\good{y_i} - \key{\hat{y}_i})^2}{\sum_i (\good{y_i} - \bar{y})^2}$$

• TSS (total sum of squares) is how much $y$ varies around its own average: the error you’d make if you ignored $x$ and always guessed the average.
• RSS (residual sum of squares) is the error left over after using the line.

For the five months, with RSS $= 161.5$:

$$\risk{\text{TSS}} = 13.4^2 + 8.6^2 + 4.4^2 + 8.6^2 + 0.6^2 = 347.2$$ $$R^2 = 1 - \frac{161.5}{347.2} = 0.535$$

The line explains $\key{53\%}$ of the month-to-month variation in volatility. Real market data is noisy; nothing here will score in the nineties. $R^2 = 0$ means no better than guessing the average. $R^2 = 1$ means perfect.

More than one input

A month is more than its volatility. Markets that have just fallen tend to stay nervous, so add last month’s return as a second input:

$$\key{\hat{y}} = \key{\hat{\beta}_0} + \key{\hat{\beta}_1} x_1 + \key{\hat{\beta}_2} x_2$$

Now the “line” is a flat surface in three dimensions, but the idea is the same: find the numbers that make the squared misses smallest.

Trained on the 4,930 rows of 2020 to 2023 (one per ticker per trading day, all five tickers), then tested on 2024 to 2025 (Table 3):

Reading $\key{\hat{\beta}_2}$: a month that fell 10% instead of staying flat means $-0.112 \times -10 = 1.1$ more points of next-month volatility, with this month’s volatility held the same. That last part matters. In multiple regression, each number is the effect of one input while the others stay fixed. The sign is right (falls make markets nervous), and the size is sensible.

Now look at the last row. The second input improved the fit on the training rows, as adding an input always does, and did nothing for the fit on the test rows: $R^2$ went from 0.454 to 0.453. The return carries a little information about 2020 to 2023, but whatever the model learned through it didn’t repeat. More inputs are not automatically better; only the test score can tell you. That tension gets a whole post, bias and variance.

(The test $R^2$ sits above the training one, 0.454 against 0.424. That looks backwards until you remember that $R^2$ depends on the data as much as on the model: 2024 to 2025 were calmer than 2020 to 2023, and a calm test period scores higher than a wild one whatever the model does.)

The matrix form

With many inputs, the formula is written with matrices. Put the inputs in a matrix $X$ (one row per day, plus a column of 1s for the intercept) and the answers in a vector $y$. The best coefficients are:

$$\key{\hat{\beta}} = (X^\top X)^{-1} X^\top y$$

This comes from the normal equations $X^\top X \key{\hat{\beta}} = X^\top y$ (Murphy, Probabilistic Machine Learning, chapter 11). ISLP doesn’t print the matrix form, because it needs matrix algebra. In practice, libraries solve it with more numerically stable methods than inverting the matrix, but the answer is the same. On the five months it gives exactly $(-12.7,\ 1.63)$.

The code

The five months from the worked example, first by hand with the formulas above, then with scikit-learn. It runs as pasted (pip install numpy scikit-learn):

import numpy as np
from sklearn.linear_model import LinearRegression

x = np.array([20, 13, 24, 16, 17])         # this month's volatility, % a year
y = np.array([30, 8, 21, 8, 16])           # next month's

# By hand: the slope and intercept formulas above
b1 = ((x - x.mean()) * (y - y.mean())).sum() / ((x - x.mean()) ** 2).sum()
b0 = y.mean() - b1 * x.mean()
print(b0, b1)                               # -12.71  1.63

# With scikit-learn: same answer
X = x.reshape(-1, 1)                        # a table: 5 rows, 1 column (scikit-learn wants a table)
model = LinearRegression().fit(X, y)
print(model.intercept_, model.coef_)        # -12.71  [1.63]: one slope per input column
print(model.predict([[20]]))                # [19.86]: one new row, with one input of 20

Python prints the numbers unrounded (-12.714285714285712 1.6285714285714286); the comments show them rounded. x is a plain list of five numbers, but scikit-learn always expects a table of rows and columns, even when there is only one input: reshape(-1, 1) turns it into five rows of one column, and [[20]] is a table of one row and one column.

How sure is a coefficient?

A fitted slope is an estimate from one sample of data, and a different four years would give a different number. The standard error is how far the estimate would typically move between samples, and dividing the slope by it gives a t-statistic: how many standard errors the slope sits from zero. About 2 is the usual bar for “probably not zero”, and the slope plus or minus two standard errors is a rough 95% interval.

For the one-input model on SPY’s training rows (this month’s volatility predicting next month’s, 986 trading days), the estimates are in Table 4:

The slope is eighteen standard errors from zero: this month’s volatility really does predict next month’s. Add yesterday’s one-day return as a second input and its coefficient is −1.22 with a standard error of 0.24, a $t$ of −5.0, also real but small, and $R^2$ rises only from 0.243 to 0.262. A low $R^2$ with a huge $t$ is the normal shape in finance: the relationship is real and the noise is large.

Reading the residuals

The misses, actual minus predicted, are the model’s confession. Plot them two ways, as in Figure 2:

Of the patterns in Table 5, the fan and the runs are both in this data. The residuals’ spread is 4.2 points in the calmest fifth of predictions and 16.5 in the wildest fifth, a fan. And 96% of consecutive residuals share a sign, where independent misses would share one half the time, because adjacent months overlap in the days they are computed from. Neither pattern stops the model working; both say its error bars are wider and its months fewer than they look.

Two cousins of least squares live in later posts. Ridge and lasso add a penalty on the size of the coefficients, which tames the “inputs that move together” problem below; the features post shows both. Logistic regression squashes the same weighted sum into a probability for a yes-or-no label; the scikit-learn post uses it on the stress-month problem.

When linear regression goes wrong

• The relationship isn’t a straight line. If volatility only matters above some level, a line will miss it. (Decision trees, later in the series, handle this.)
• Extrapolation. Never trust the line far outside the data, like the “−12.7% next month after a month with no volatility”.
• Outliers. Because misses are squared, one extreme month (March 2020) can pull the line a long way. The five-month example avoided it; the four-year fit had to live with it.
• Inputs that move together. If two inputs are strongly linked (5-day and 20-day volatility, say), the model can’t tell their effects apart, and the individual numbers become unstable, even when the predictions are fine.
• Correlation is not causation. The slope says this month’s and next month’s volatility move together. It doesn’t say one causes the other; both may be reacting to the same news.

Summary

Table 6 is the post in six rows.

References

• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), sections 3.1 to 3.3: the model (equations 3.1, 3.2), RSS (3.3), least-squares estimates (3.4), $R^2$ (3.17)
• The same book, section 3.1.2: standard errors, the “plus or minus two standard errors” interval (equation 3.9) and the t-statistic (3.14); section 3.3.3: potential problems, including a curve the line misses, misses that run in the same direction over time, noise that grows, outliers and inputs that move together
• Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), section 11.2.2.1: ordinary least squares and the normal equations
• scikit-learn: ordinary least squares