Bias and variance: where a model's errors come from

This is the idea behind overfitting, and it’s explained in section 2.2.2 of An Introduction to Statistical Learning. It’s the reason a model that looks perfect in testing can fail in production, and the reason three later posts (bagging, random forests, boosting) exist at all.

In the decision trees post, a fully grown tree scored a perfect 1.000 on the rows it trained on and 0.742 on test rows it had not seen (on a scale where 0.5 is a coin flip). This post explains why, with a formula you can actually measure.

The idea, with a person

Imagine two new risk analysts forecasting next month’s volatility.

• Analyst A has one rule: “a jumpy month is followed by a jumpy month.” Simple, consistent, and often wrong in the same way. That’s bias: a model too simple for the real pattern.
• Analyst B remembers every single month they’ve ever seen and predicts from the most similar one. Give B a different few years of history, and B’s forecasts change completely. That’s variance: a model that reacts too much to the particular data it learned from.

A good analyst is in between: they learn the real pattern and ignore the one-off quirks.

The formula

For a model $\key{\hat{f}}$ trained on some data, the expected squared error on a new point $x_0$ splits into three parts. This is equation 2.7 in An Introduction to Statistical Learning:

f̂, the model's predictionf and y, the truthbiasvariance

Written out term by term (same thing, with each part spelled out; the symbols are in Table 1):

In words:

• Bias$^2$: how far the model’s average guess is from the truth. Too simple → large.
• Variance: how much the model’s guess jumps around depending on the sample it trained on. Too complex → large.
• Noise: the part of the error that stays even with a perfect model.

The expectation $E[\cdot]$ here is over many different training samples. That is the thing to hold on to about variance. It is not a property of one trained model. It measures how much the model would change if you had collected different data.

Measuring it

Usually you can’t measure bias and variance, because you don’t know the true pattern $\good{f(x)}$. With made-up data, you can. I invented a true curve for next month’s volatility as a function of this month’s, shaped like the real SPY data in the linear regression post: a gentle slope, plus a step up once the market is already above about 25% (once markets are wild, they tend to stay wild). Then I added random noise with a standard deviation of 3 points (so $\sigma^2 = 9$).

Then, for each tree depth, I repeated this 300 times: draw 80 fresh months, train a regression tree, record its predictions. That gives me 300 versions of each model, enough to measure its average and its spread (Table 2).

Two things to see. First, the formula works: the three parts add up to the error I measured directly on new months, to within simulation noise. Second, the trade-off, drawn in Figure 1:

• Depth 1 (blue on the right): a single question. Every version gives nearly the same flat steps, and all of them miss the shape of the curve. High bias.
• Depth 15 (red): every version zig-zags through its own noise, and no two agree. Bias is almost zero, because on average they follow the curve, but each one alone is far off. High variance.
• Depth 3 gives the lowest total error here. Not the most flexible model, not the simplest. Three questions are enough to place the step and the slope on either side of it.

A detail I did not expect: depths 1 and 2 have more variance than depth 3. With so few splits, where the cut-offs land jumps around a lot between samples. The general trend (more depth, more variance) holds after that.

Overfitting and underfitting

These are the everyday names (Table 3):

The tell-tale sign of overfitting is a big gap between training error and error on new data, like the decision tree with a perfect 1.000 on training rows and 0.742 on test rows.

The same curve on real data

On the real version of the problem, predicting next month’s volatility from this month’s for the five tickers, with one regression tree and the minimum rows per leaf swept from 1 to 200 (Table 4). The score is RMSE, the typical size of a miss in points of volatility; lower is better:

Training error rises the whole way. Test error falls the whole way too, and at 200 rows per leaf (a fifth of the training set) it is still falling, because this target is so noisy that a tree needs to average hundreds of rows before it says anything true. The right-hand side of the U, where a too-simple tree blurs the real pattern, lies past the sizes swept here; the bagging post shows what happens when averaging, not leaf size, does the smoothing. Read any model from the two numbers (Table 5):

Two rules of thumb that go with the curve. Choose the simplest model whose validation error is within one standard error of the best (ISLP section 6.1.3 calls this the one-standard-error rule; the standard error is how much the validation score itself would wobble with different data), since a difference that small is noise. And remember that some dials run backwards: in nearest neighbours a larger $k$ is a simpler model, and in ridge or lasso a larger penalty is a simpler model, the opposite of tree depth and polynomial degree.

How to find the sweet spot in practice

You never know the true curve in real life, so you can’t compute bias and variance directly. Instead:

• Hold out data. Train on part of the history, test on the rest. Only the test score counts.
• Cross-validation. Split the data into, say, 5 parts. Train on 4, test on the 5th, rotate, and average. More reliable than one split.
• For time series, test forward in time. In finance, never test on dates before your training data. Train on January to June, test on July. Otherwise the model quietly learns from the future.
• Tune complexity on validation data, a slice held back from training, or on cross-validation: tree depth, number of features, and so on. Keep the real test set for one final check at the end. A setting chosen by looking at the test score makes that score look better than the model is, the subject of the scoring post.

And there’s a third option, which the bagging post builds on: keep the complex, low-bias model, but reduce its variance by averaging many of them. If each deep tree zig-zags differently, the average of many comes much closer to the real curve. That’s bagging.

The code

The simulation behind Table 2, complete, so it runs as pasted (pip install numpy scikit-learn). true_curve is the made-up truth; each round draws 80 fresh months (x is this month’s volatility, y next month’s), trains one tree, and records its predictions at 200 test points. preds ends up with 300 rows, one per trained tree, which is what makes bias and variance measurable:

import numpy as np
from sklearn.tree import DecisionTreeRegressor

def true_curve(x):
    """A made-up 'true' next-month volatility for this month's volatility x (both % a year):
    a gentle slope, plus a step up once the market is already above about 25%."""
    return 6 + 0.45 * x + 8 / (1 + np.exp(-(x - 25) / 3))

NOISE = 3.0                              # spread of the random part no model can predict
grid = np.linspace(5, 45, 200)           # 200 values of this month's volatility to test at
truth = true_curve(grid)                 # known only because we made the curve up
rng = np.random.default_rng(11)

print("depth  bias^2  variance  noise  sum    measured error on new months")
for depth in [1, 2, 3, 4, 5, 6, 8, 10, 15]:
    preds, errors = [], []
    for _ in range(300):                 # 300 different training samples of 80 months each
        sample = np.random.default_rng(rng.integers(1 << 31))
        x = sample.uniform(5, 45, 80)                        # this month's volatility
        y = true_curve(x) + sample.normal(0, NOISE, 80)      # next month's: truth plus noise
        tree = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(x.reshape(-1, 1), y)
        p = tree.predict(grid.reshape(-1, 1))
        preds.append(p)
        y_new = truth + rng.normal(0, NOISE, len(grid))     # fresh months the tree never saw
        errors.append(np.mean((y_new - p) ** 2))
    preds = np.array(preds)                                  # 300 rows: one per trained tree
    bias2 = np.mean((preds.mean(axis=0) - truth) ** 2)       # how far the AVERAGE tree is off
    variance = np.mean(preds.var(axis=0))                    # how much the trees disagree
    print(f"{depth:5}  {bias2:6.2f}  {variance:8.2f}  {NOISE**2:5.2f}  "
          f"{bias2 + variance + NOISE**2:5.2f}  {np.mean(errors):5.2f}")

It prints Table 2, number for number, including the depths left out of it (4, 6 and 10). preds.mean(axis=0) averages the 300 trees at each test point (down the rows), and preds.var(axis=0) measures how much they disagree there.

Summary

Table 6 is the post in six lines.

References

• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 2.2.2: the bias-variance trade-off (equation 2.7); section 6.1.3: the one-standard-error rule
• scikit-learn: cross-validation