Choosing a model is a systems decision

The series has now built a linear model, nearest neighbours, a tree, a forest and a boosted model on the same problem. This post puts them in one table and adds the columns that accuracy leaves out, because those columns decide which model gets deployed.

the chosen model or its predictionerror, or a cost

Five models, one table

Next month’s volatility from this month’s prices, fitted on 2020 to 2023, scored on 2024 to 2025, timed on one laptop core. Test RMSE is the typical size of a miss in points of volatility; lower is better. The last two columns of Table 1 time a single prediction: the median, and the worst one in every hundred:

Read it column by column.

Accuracy. The spread is half a volatility point (the first panel of Figure 1), and the baseline is five points behind the worst model. Nearest neighbours wins the column by a hair over the forest, the tree and boosting sit within 0.1 of it, and the line is last. The signal is weak, and the ordering is not the one the models’ reputations predict; on the stress-month classification in the forests post it was different again. The accuracy column has to be measured, not assumed. The line is highlighted, not the winner, because the rest of the table is the decision.

Training time. The forest took 2 seconds, about 700 times the linear model. That only matters against a budget: a 40-minute fit inside a four-hour overnight window is free; the same fit every time a feature changes is not.

Prediction time, and its tail. The linear model answers a question in 30 microseconds; the forest in 5.9 milliseconds, because every prediction walks 300 trees. Nearest neighbours is cheap to train and the only model whose prediction cost grows with the data, because it scans every stored row. The last column is the one a trading deadline cares about: not the average latency but the worst one in a hundred. A mean of 2 ms inside a 10 ms deadline looks safe, but if the worst one in a hundred takes 40 ms, then at least one order in a hundred misses the deadline.

What it keeps. Nearest neighbours keeps every training row forever, which is a memory cost and sometimes a privacy or licensing one. The others keep parameters: seven numbers for the linear model, 19 leaves for the tree, 36,397 leaves for the forest.

Interpretability. The linear model’s seven coefficients can be read aloud: more 20-day volatility means more next-month volatility. Its coefficient is 678 per unit because the feature is measured as a daily standard deviation: a rise of 0.01 (about 16 points a year) adds about 6.8 points to next month’s forecast. The tree with 19 leaves can be drawn on a page. The forest cannot, and a model that cannot be explained cannot be audited, which in some settings makes it unusable whatever its accuracy. Partial-dependence curves (how the prediction moves as one feature changes, the others held fixed) and importance scores recover some of it, at a cost.

Extrapolation: the question a test set never asks

The test rows all lie within the range of the training rows. Ask each model about a month twice as volatile as any it has seen, with the recent-volatility features doubled from the wildest training row (Table 2):

Every tree-based model and the neighbour model answer with roughly what followed the wildest month they have ever seen, not the 138 that month itself reached, because a tree’s leaf is a constant and a neighbour’s answer is an average of stored rows. The linear model follows its slope off the edge of the map. Neither is right; the point is that they fail differently, and that a model’s behaviour outside its training range is a design choice, not something a test score reveals. Finance meets the outside of the range on the worst day of the decade.

Stability

A linear model refitted on slightly different data gives nearly the same line. A single tree can flip its root split and everything below it; that instability is why the bagging post exists. A forest is stable again by averaging. Stability across random seeds is cheap to measure and worth reporting next to the score; it is not the same as holding up when the market itself changes, which only testing forward in time shows.

The comparison table

Worked costs for the cost-per-prediction row of Table 3: a table of one million rows and twenty features. One nearest-neighbour prediction touches twenty million numbers; one linear prediction touches twenty; a depth-12 tree makes twelve comparisons; a 500-tree forest of them makes six thousand.

How to decide

1. Write down the constraints before the accuracy: latency budget and its tail, how often the model refits, whether a human must be able to explain a prediction, how much data there is per feature.
2. Measure the baseline, then the simplest model that fits the constraints. Linear first.
3. Move up only when the gain is bigger than the noise between seeds and worth its extra cost.
4. Check the behaviour outside the training range on purpose.
5. Prefer the model whose failures you understand.

There is no best model for every dataset. There is a best model for this dataset under these constraints, and the constraints are most of the decision.

Traps

Table 4 lists the traps.

The code

Four of the models on made-up data, so it runs as pasted (pip install numpy scikit-learn). X_train holds this month’s volatility as a one-column table, y_train next month’s. models is a dictionary of name to model, so one loop fits, times and questions every model the same way.

import time
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.neighbors import KNeighborsRegressor
from sklearn.tree import DecisionTreeRegressor

# Made-up data: this month's volatility predicts next month's, nearly in a straight line
rng = np.random.default_rng(0)
x = rng.uniform(5, 45, 3000)
y = 6 + 0.6 * x + rng.normal(0, 4, 3000)
X_train, y_train = x[:2000].reshape(-1, 1), y[:2000]       # tables of one column
X_test = x[2000:].reshape(-1, 1)

models = {"linear": LinearRegression(),
          "nearest neighbours": KNeighborsRegressor(100),
          "one tree": DecisionTreeRegressor(min_samples_leaf=200),
          "random forest": RandomForestRegressor(300, min_samples_leaf=20, random_state=0)}

extreme = [[90]]                                   # twice the wildest month in training (45)
for name, model in models.items():
    t0 = time.perf_counter(); model.fit(X_train, y_train); fit_ms = (time.perf_counter() - t0) * 1000
    singles = []
    for row in X_test[:200]:                       # time 200 single-row predictions
        t0 = time.perf_counter(); model.predict(row.reshape(1, -1))
        singles.append((time.perf_counter() - t0) * 1000)
    p50, p99 = np.percentile(singles, [50, 99])    # the median, and the worst 1 in 100
    print(f"{name:18} fit {fit_ms:7.1f} ms | one row: median {p50:.3f} ms, worst 1 in 100 "
          f"{p99:.3f} ms | asked about 90: {model.predict(extreme)[0]:.0f}")

It prints one line per model: the fit time, the median and worst-in-a-hundred time for one prediction, and its answer for a month twice as wild as any in training. The forest is by far the slowest to ask (several milliseconds a row, against hundredths for the line). Asked about the extreme month, the line follows its slope to about 60, while the tree, the forest and nearest neighbours stay near 31 to 33, the most extreme values they have seen. Times depend on your machine; the order does not. time.perf_counter() is Python’s most precise clock.

Summary

Table 5 is the post in seven lines.

References

• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 2.1.3: the trade-off between prediction accuracy and how easy a model is to interpret
• Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), section 1.2.4: the no free lunch theorem, no single model is best for every dataset
• scikit-learn: computational performance, prediction latency and throughput