Random forests: bagging with one extra rule

Random forests are covered in section 8.2.2 of An Introduction to Statistical Learning. The method is Leo Breiman’s, from his 2001 paper “Random Forests”. It works well with little tuning on tables of features, the shape most finance data comes in: credit applications, payments, daily prices.

The bagging post ended on a limit. Averaging many trees cuts variance only down to a floor:

ρ, the correlation between the trees' errorsσ², the variance of one tree

$$\text{floor} = \key{\rho}\,\risk{\sigma^2}$$

$\key{\rho}$ is the correlation between the trees’ errors: 1 means they all miss in the same way, 0 means their misses are unrelated. $\risk{\sigma^2}$ is the variance of a single tree, how much it changes when it’s trained on different data. Averaging shrinks the second term towards zero but can’t touch the first. Random forests attack $\key{\rho}$.

The problem with bagging: every tree looks the same

On the stress-month data, the strongest single signal of a rough month ahead is the 20-day volatility of the month just gone. I bagged 300 trees and looked at the first question each tree asks:

(The other 7 start with the 5-day volatility, which says nearly the same thing.) Each tree sees a different bootstrap sample, but the signal is so strong that almost every tree starts with it, and probably makes similar choices after it. The trees are different, but not very different. Their mistakes are correlated, so averaging them can’t cancel much.

The fix: limit the choice at each split

A random forest adds one rule to bagging:

At every split, pick a random subset of $\key{m}$ features, and only consider those.

Sometimes the 20-day volatility isn’t in the subset. Then the tree has to find the next-best question using the other features, and it learns something the other trees didn’t.

Now the trees genuinely differ: some start with the 5-day volatility, some with the 20-day return, some with the price against its moving average. Each is a bit weaker on its own, but they are wrong in more different ways, so their average is better.

One split, with seven features and $\key{m} = 3$:

How big should $\key{m}$ be? With $p$ features, the usual starting point for classification is

m, the features each split may use

$$\key{m} \approx \sqrt{p}$$

which is what ISLP recommends and what scikit-learn’s RandomForestClassifier uses by default. With seven features, $\sqrt{7} \approx 2.6$, so the default is 2; the forests above used 3 and 1.

Note. scikit-learn’s RandomForestRegressor defaults to all features, which is plain bagging. Set max_features yourself, and tune it.

These are starting points, not laws: tune $\key{m}$ on held-out data. Setting $\key{m} = p$ turns a random forest back into plain bagging.

The results

Same 3,685 training rows (2019 to 2021) and 2,405 test rows (2022 to 2023) as in the earlier posts. Every tree is grown until its leaves hold 20 rows, so the only difference between the models in the table is the randomness. Scores are AUC on the test rows (0.5 is a coin flip, 1.0 is perfect):

Are gaps this small real, or just the seed? I re-ran each ensemble with five random seeds: bagging scored 0.808–0.809, the forest with 3 of 7 features 0.817–0.820, with 1 of 7 features 0.820–0.821, and extra trees 0.825–0.826. The ranges don’t overlap, so the order holds; only the two forests are close enough to call a near tie.

Each step adds more randomness, and each step helps. The last row is a variation called extremely randomised trees (scikit-learn’s ExtraTreesClassifier). It also picks the cut-off values at random instead of searching for the best one, which makes the trees even more different and very fast to train.

With only seven features, several of which say similar things, the gains are modest. Random forests shine most with many features, where a few strong ones would otherwise dominate every tree.

Which features matter?

Random forests can report how much each feature contributes. There are two common ways, and they can disagree badly.

Impurity importance (scikit-learn’s feature_importances_): add up how much each feature reduced Gini impurity across all the splits in all the trees. Gini impurity measures how mixed the classes are in a node: 0 means every row in it has the same outcome. It’s free, since it comes from training.

Permutation importance: take the test set, randomly shuffle one feature’s column so it no longer carries any information, and measure how much the test score drops. The bigger the drop, the more the model relied on that feature.

The two columns are in different units (share of all impurity reduction vs. drop in test AUC), so compare the rankings, not the values.

Look at 5-day volatility. Impurity importance gives it a quarter of all the credit, a solid second place. Permutation importance says shuffling it costs almost nothing: 0.022, an eighth of the 20-day figure. Neither is wrong. The gap between them is what matters.

Why do they disagree? The 5-day and 20-day volatility are strongly correlated; they measure the same thing over different windows. Inside the forest the trees split on both, so both earn impurity credit. But when you shuffle the 5-day column, the trees that leaned on it are still fed the 20-day column, which carries nearly the same information, and the forest barely notices. Impurity importance says “how often was it used”; permutation importance says “how much would we miss it”. For anything you’ll act on, the second question is the one you’re asking, so use permutation importance on test data. (scikit-learn’s own documentation adds a second warning: impurity importance is also biased towards features with many distinct values.) A negative permutation score, like the Bollinger position’s, just means noise: shuffling it changed nothing and the test score wobbled.

The code

Made-up data with seven features, like the series’ seven, so it runs as pasted (pip install numpy scikit-learn). vol_20d carries the signal, vol_5d is its near-twin, and the other five carry nothing. X holds the features, one row per case; y the answer, 1 if a stress month followed. The first 3,000 rows train, the last 1,000 test.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.metrics import roc_auc_score

# Made-up data: seven features, like the series' seven
rng = np.random.default_rng(0)
n = 4000
vol_20d = rng.uniform(5, 50, n)
vol_5d = vol_20d + rng.normal(0, 4, n)          # a near-twin of vol_20d
others = rng.normal(size=(n, 5))                 # five features that carry nothing
X = np.c_[vol_20d, vol_5d, others]
names = ["vol_20d", "vol_5d", "f3", "f4", "f5", "f6", "f7"]
y = (vol_20d / 50 + rng.normal(0, 0.2, n) > 0.75).astype(int)   # 1 = stress month ahead
X_train, X_test, y_train, y_test = X[:3000], X[3000:], y[:3000], y[3000:]

# max_features=None lets every split look at all 7 features: that is plain bagging
for name, m in [("bagging (all 7)", None), ("forest, 3 of 7  ", 3), ("forest, 1 of 7  ", 1)]:
    model = RandomForestClassifier(300, max_features=m, min_samples_leaf=20,
                                   random_state=0, n_jobs=-1).fit(X_train, y_train)
    first = [t.tree_.feature[0] for t in model.estimators_]      # each tree's first question
    print(f"{name}: {first.count(0)} of 300 trees start with vol_20d, "
          f"test AUC {roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]):.3f}")

# Two ways to rank features, on the last forest (1 of 7 per split)
perm = permutation_importance(model, X_test, y_test, scoring="roc_auc", n_repeats=10, random_state=0)
for i in range(2):
    print(f"{names[i]}: impurity importance {model.feature_importances_[i]:.2f}, "
          f"drop in test AUC when shuffled {perm.importances_mean[i]:.3f}")

It prints, for bagging, 300 of 300 trees starting with vol_20d; for the forests, about 140 and 45. That is the same effect as the real table above: limiting the choice forces the trees to differ. On this toy data the three AUCs come out about the same (about 0.89); on the real data the gain was small too, 0.809 to 0.821. Then the two importances disagree on the twin, as on the real data: impurity importance gives vol_5d about 0.38, but shuffling it costs only about 0.05 of AUC, because vol_20d carries the same information. t.tree_.feature[0] is the feature a tree asks about at its top node.

The script behind the real numbers

The tables come from examples/random-forests/random_forest.py, which runs on the blog’s five tickers. To run it yourself, clone the examples repo: its README has a three-command Docker setup, or run uv run random-forests/random_forest.py.

Why banks like random forests

• Few settings to get wrong. The defaults are usually decent.
• Adding more trees doesn’t overfit. It just stops improving (ISLP section 8.2.2: “random forests will not overfit if we increase B”, the number of trees). Each tree can still overfit if it is grown deep on small data.
• Handles mixed data (yes/no flags, amounts, counts) without scaling.
• Built-in honesty check: out-of-bag scores, as in bagging (with the same time-series caveat).

And the trade-offs: a forest of 300 trees is much harder to explain to a regulator or a business user than one tree, and it’s slower to score, which is why the importance analysis above matters.

Summary

References

• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 8.2.2: random forests, $m \approx \sqrt{p}$, and why more trees do not overfit
• Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), section 18.4: random forests decorrelate the trees with a random subset of features at each split
• Leo Breiman, “Random Forests”, Machine Learning 45(1), 5–32, 2001
• Pierre Geurts, Damien Ernst and Louis Wehenkel, “Extremely randomized trees”, Machine Learning 63(1), 3–42, 2006
• scikit-learn: permutation importance vs impurity-based importance