Scoring a model: a number means nothing without a split, a metric and a baseline

“The model scores 0.78.” That sentence contains no information until three questions are answered: scored on what rows, by what measure, and compared with what. This post is those three questions, worked on the series’ two problems: predicting next month’s volatility (a number) and predicting whether next month is a stress month (a yes or no).

a prediction or a chosen settingan errorthe actual valuea miss the model will regret

Which rows: the three roles

Every row of data plays one of three roles (Table 1):

Change anything after seeing the test score and it is no longer a test score; it has become a second validation set, and the number it gives you is optimistic by an amount you cannot know. After choosing the settings you may refit on training plus validation, then score the test set.

For a time series the test must be later than the training. The series splits at the end of 2023: fit on 2020 to 2023, score on 2024 to 2025. A random split would put near-twins of every test row into the training set (two rows a day apart are computed from almost the same month of prices) and flatter every model. Cross-validation does the same job with several folds, as in the data post; for time series the folds move forward, never backward.

Which measure: three ways to add up misses

A regression model’s misses are $\good{y_i} - \key{\hat{y}_i}$. Three ways to turn a list of them into one number:

$$\text{MAE} = \frac{1}{n}\sum_i \lvert \good{y_i} - \key{\hat{y}_i} \rvert$$ $$\text{MSE} = \frac{1}{n}\sum_i (\good{y_i} - \key{\hat{y}_i})^2$$ $$\risk{\text{RMSE}} = \sqrt{\text{MSE}}$$

Mean absolute error treats every miss by its size. Mean squared error squares each miss first, so one big miss dominates. Root mean squared error puts the squared version back into the units of the target. Worked on four misses of 1, 1, 1 and 5 (Table 2):

RMSE is never below MAE, and they are equal only when every miss is the same size. A big gap between them is a sign of a few large misses. Choose by the cost shape: if a miss twice as large costs twice as much, use MAE; if it costs far more than twice, use RMSE. Squared error is also the one that chases outliers when fitting, which is why a crash day can tilt a whole model.

Real run. Next month’s volatility on the 2,205 test rows (one per ticker per trading day, 2024 to 2025), a linear model against the baseline of always predicting the training mean (Table 3):

The RMSE sits well above the MAE for the linear model, because a few rows were missed by a lot: the largest miss is 54.7 points.

Compared with what: the baseline

The first panel of Figure 1 has a dotted horizontal line at 29.8, the training mean. Every model has to beat that line, and in the ten-ideas post a fully grown tree did not: it scored an RMSE of 18.26 against the baseline’s 17.69, having memorised the training rows. A number like 18.26 looks fine in isolation and is worse than doing nothing.

For the stress-month problem, the baseline in Table 4 is the majority class: 75.9% of test rows have a calm month ahead, so “always say calm” scores 75.9% accuracy. A model reporting 76% accuracy has learned almost nothing.

Classification: the confusion matrix

A classifier first produces a probability, then a threshold turns it into a yes or no. The four outcomes (Table 5):

Four rates come out of it (Table 6), and the way to keep them straight is what they divide by:

Recall and specificity divide by the true counts; precision divides by the predicted count. The false alarm rate is one minus specificity.

Real run. A random forest on the 2,205 test rows, 532 of which had a stress month ahead. Same probabilities, three thresholds (Table 7):

Accuracy ranks the 0.5 row best and falls with every step down. But at 0.5 the model misses 165 of the 532 stress cases; at 0.3 it misses 126 and raises 107 more false alarms. Which row is right depends on what a miss costs. If a missed stress month means an unhedged loss and a false alarm means a few days of unnecessary hedging, the lower threshold is the better model, and accuracy ranked it worse. Move the threshold to match the costs, and report the rate that matches the question.

ROC and AUC: every threshold at once

Slide the threshold from 1 down to 0 and plot recall against the false alarm rate at each step. That curve is the ROC curve, the third panel of Figure 1. A model with no skill runs along the diagonal; a perfect one goes straight up the left edge. The area under it, AUC, summarises the curve: 0.5 is a coin flip, 1.0 is perfect. The forest scores $\key{0.894}$. AUC is threshold-free, which makes it the right single number for comparing models; it is the wrong number for deciding what to do, because doing something needs a threshold.

The fair-comparison checklist

Two scores are comparable only if they share all of these:

• the same rows, over the same period, with the test later than the training;
• the same metric in the same units (a percentage point is 0.01 in decimal returns);
• preprocessing (scaling, filling, feature selection) fitted on the training part only;
• several runs, with the spread reported, not the best run.

One run is one draw. A score that holds steady across random seeds can still break when the market itself changes, as 2020 showed.

Traps

Table 8 lists the traps.

The code

The four misses from Table 2, then a classifier on made-up data where stress is rare, so it runs as pasted (pip install numpy scikit-learn). X holds one feature, this month’s volatility; stress holds the answer, 1 if a stress month followed. The first 2,000 rows train, the last 1,000 test.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix, roc_auc_score

# 1. Three ways to add up misses: the four misses from the table above
e = np.array([1, 1, 1, 5])                       # actual minus predicted, one per row
print("MAE", np.abs(e).mean(), " MSE", (e ** 2).mean(), " RMSE", round(np.sqrt((e ** 2).mean()), 2))

# 2. A classifier, its threshold and its AUC, on made-up data where stress is rare
rng = np.random.default_rng(0)
n = 3000
vol = rng.uniform(5, 50, n)                      # one feature: this month's volatility
stress = (vol / 50 + rng.normal(0, 0.25, n) > 0.75).astype(int)   # 1 = stress month ahead, rare
X = vol.reshape(-1, 1)                           # a table: one row per case, one column
X_train, X_test = X[:2000], X[2000:]
y_train, y_test = stress[:2000], stress[2000:]
print("share of test rows with stress ahead:", round(y_test.mean(), 3))

clf = RandomForestClassifier(300, min_samples_leaf=20, random_state=0).fit(X_train, y_train)
prob = clf.predict_proba(X_test)[:, 1]           # the model's chance of stress, one per row
for threshold in (0.5, 0.3, 0.2):
    said_stress = prob >= threshold              # True where the chance is above the threshold
    tn, fp, fn, tp = confusion_matrix(y_test, said_stress).ravel()
    print(f"threshold {threshold}: accuracy {(tp + tn) / len(y_test):.2f}, "
          f"recall {tp / (tp + fn):.2f}, precision {tp / (tp + fp):.2f}")
print("AUC", round(roc_auc_score(y_test, prob), 3))   # one number for every threshold at once

It prints MAE 2.0 MSE 7.0 RMSE 2.65, Table 2 again. Then, for the made-up classifier, the same pattern as the real one: as the threshold drops from 0.5 to 0.2, recall rises (about 0.57 to 0.87) while precision and accuracy fall. Then an AUC of about 0.84. .ravel() flattens the $2 \times 2$ confusion matrix into its four counts, in the order TN, FP, FN, TP.

The script behind the real numbers

The tables above come from examples/scoring-a-model/scoring.py, which runs on the blog’s five tickers. To run it yourself, clone the examples repo: its README has a three-command Docker setup, or run uv run scoring-a-model/scoring.py.

Summary

Table 9 is the post in seven lines.

References

• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 2.2: assessing accuracy; section 4.4.2: the confusion matrix, changing the threshold, and the ROC curve
• scikit-learn: metrics and scoring, the definitions behind every function used here
• scikit-learn: common pitfalls and recommended practices, on data leakage into test scores