Random forests: bagging with one extra rule

Random forests are covered in section 8.2.2 of An Introduction to Statistical Learning. The method is Leo Breiman’s, from his 2001 paper “Random Forests”. In banking it is probably the most used model for tabular data, because it works with little tuning: credit scoring, fraud, operations risk.

The bagging post ended on a limit. Averaging many trees cuts variance only down to a floor:

ρ, the correlation between the trees' errorsσ², the variance of one tree

$$\text{floor} = \key{\rho}\,\risk{\sigma^2}$$

$\key{\rho}$ is the correlation between the trees’ errors: 1 means they all miss in the same way, 0 means their misses are unrelated. $\risk{\sigma^2}$ is the variance of a single tree, how much it changes when it’s trained on different data. Averaging shrinks the second term towards zero but can’t touch the first. Random forests attack $\key{\rho}$.

The problem with bagging: every tree looks the same

On the stress-month data, the strongest single signal of a rough month ahead is the 20-day volatility of the month just gone. I bagged 300 trees and looked at the first question each tree asks:

(The other 7 start with the 5-day volatility, which says nearly the same thing.) Each tree sees a different bootstrap sample, but the signal is so strong that almost every tree starts with it, and probably makes similar choices after it. The trees are different, but not very different. Their mistakes are correlated, so averaging them can’t cancel much.

The fix: limit the choice at each split

A random forest adds one rule to bagging:

At every split, pick a random subset of $\key{m}$ features, and only consider those.

Sometimes the 20-day volatility isn’t in the subset. Then the tree has to find the next-best question using the other features, and it learns something the other trees didn’t.

Now the trees genuinely differ: some start with the 5-day volatility, some with the 20-day return, some with the price against its moving average. Each is a bit weaker on its own, but they are wrong in more different ways, so their average is better.

One split, with seven features and $\key{m} = 3$:

How big should $\key{m}$ be? With $p$ features, the usual starting point for classification is

m, the features each split may use

$$\key{m} \approx \sqrt{p}$$

which is what ISLP recommends and what scikit-learn’s RandomForestClassifier uses by default. With seven features, $\sqrt{7} \approx 2.6$, so the default is 2; the forests above used 3 and 1.

Note. scikit-learn’s RandomForestRegressor defaults to all features, which is plain bagging. Set max_features yourself; The Elements of Statistical Learning suggests $p/3$ for regression.

These are starting points, not laws: tune $\key{m}$ on held-out data. Setting $\key{m} = p$ turns a random forest back into plain bagging.

The results

Same 3,685 training rows (2019 to 2021) and 2,405 test rows (2022 to 2023) as in the earlier posts. Every tree is grown until its leaves hold 20 rows, so the only difference between rows is the randomness. Scores are AUC on the test months (0.5 is a coin flip, 1.0 is perfect):

Are gaps this small real, or just the seed? I re-ran each ensemble with five random seeds: bagging scored 0.808–0.809, the forest with 3 of 7 features 0.817–0.820, with 1 of 7 features 0.820–0.821, and extra trees 0.825–0.826. The ranges don’t overlap, so the order holds; only the two forests are close enough to call a near tie.

Each step adds more randomness, and each step helps. The last row is a variation called extremely randomised trees (scikit-learn’s ExtraTreesClassifier). It also picks the cut-off values at random instead of searching for the best one, which makes the trees even more different and very fast to train.

With only seven features, several of which say similar things, the gains are modest. Random forests shine most with many features, where a few strong ones would otherwise dominate every tree.

Which features matter?

Random forests can report how much each feature contributes. There are two common ways, and they can disagree badly.

Impurity importance (scikit-learn’s feature_importances_): add up how much each feature reduced Gini impurity across all the splits in all the trees. Gini impurity measures how mixed the classes are in a node: 0 means every trade in it has the same outcome. It’s free, since it comes from training.

Permutation importance: take the test set, randomly shuffle one feature’s column so it no longer carries any information, and measure how much the test score drops. The bigger the drop, the more the model relied on that feature.

The two columns are in different units (share of all impurity reduction vs. drop in test AUC), so compare the rankings, not the values.

Look at 5-day volatility. Impurity importance gives it a quarter of all the credit, a solid second place. Permutation importance says shuffling it costs almost nothing: 0.022, an eighth of the 20-day figure. Neither is wrong. The gap between them is what matters.

Why do they disagree? The 5-day and 20-day volatility are strongly correlated; they measure the same thing over different windows. Inside the forest the trees split on both, so both earn impurity credit. But when you shuffle the 5-day column, the trees that leaned on it are still fed the 20-day column, which carries nearly the same information, and the forest barely notices. Impurity importance says “how often was it used”; permutation importance says “how much would we miss it”. For anything you’ll act on, the second question is the one you’re asking, so use permutation importance on test data. (scikit-learn’s own documentation adds a second warning: impurity importance is also biased towards features with many distinct values.) A negative permutation score, like the Bollinger position’s, just means noise: shuffling it changed nothing and the test score wobbled.

The code

from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance

forest = RandomForestClassifier(
    n_estimators=300,          # number of trees
    max_features=3,            # features considered at each split (m)
    min_samples_leaf=20,       # stop splitting below 20 rows
    oob_score=True,            # score each tree on the rows it never saw
    n_jobs=-1,                 # use all CPU cores
    random_state=0,
)
forest.fit(X_train, y_train)
forest.oob_score_                                      # out-of-bag accuracy, 0.878 here (optimistic on a time series)
forest.predict_proba(X_test)[:, 1]                     # probability of a stress month

result = permutation_importance(forest, X_test, y_test, scoring="roc_auc", n_repeats=10)
result.importances_mean                                # drop in AUC per shuffled feature

Why banks like random forests

• Few settings to get wrong. The defaults are usually decent.
• Adding more trees doesn’t overfit. It just stops improving (though deep trees on small data still can).
• Handles mixed data (yes/no flags, amounts, counts) without scaling.
• Built-in honesty check: out-of-bag scores, as in bagging (with the same time-series caveat).

And the trade-offs: a forest of 300 trees is much harder to explain to a regulator or a business user than one tree, and it’s slower to score. Model risk teams usually ask for importance analysis and example explanations alongside it.

Summary

Next

Boosting: instead of training trees independently and averaging, train them one after another, each one focusing on the mistakes of the ones before.

References

• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 8.2.2
• Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), chapter 18: random forests
• Hastie, Tibshirani and Friedman, The Elements of Statistical Learning (2nd ed., Springer, 2009), chapter 15
• Leo Breiman, “Random Forests”, Machine Learning 45(1), 5–32, 2001
• Pierre Geurts, Damien Ernst and Louis Wehenkel, “Extremely randomized trees”, Machine Learning 63(1), 3–42, 2006
• scikit-learn: permutation importance vs impurity-based importance