Scoring a model: a number means nothing without a split, a metric and a baseline

“The model scores 0.78.” That sentence contains no information until three questions are answered: scored on what rows, by what measure, and compared with what. This post is those three questions, worked on the series’ two problems: predicting next month’s volatility (a number) and predicting whether next month is a stress month (a yes or no).

a prediction or a chosen settingan errorthe actual valuea miss the model will regret

Which rows: the three roles

Set	Used for	How often you may look
Training	fitting the parameters	as much as you like
Validation	choosing settings: depth, $k$ , λ, which features	many times
Test	one final, honest check	once

Change anything after seeing the test score and it is no longer a test score; it has become a second validation set, and the number it gives you is optimistic by an amount you cannot know. After choosing the settings you may refit on training plus validation, then score the test set.

For a time series the test must be later than the training. The series splits at the end of 2021: fit on 2019 to 2021, score on 2022 to 2023. A random split would put near-twins of every test month into the training set (consecutive days share most of a month) and flatter every model. Cross-validation does the same job with several folds; for time series the folds move forward, never backward.

Which measure: three ways to add up misses

A regression model’s misses are $\good{y_i} - \key{\hat{y}_i}$ . Three ways to turn a list of them into one number:

\text{MAE} = \frac{1}{n}\sum_i \lvert \good{y_i} - \key{\hat{y}_i} \rvert

\text{MSE} = \frac{1}{n}\sum_i (\good{y_i} - \key{\hat{y}_i})^2

\risk{\text{RMSE}} = \sqrt{\text{MSE}}

Mean absolute error treats every miss by its size. Mean squared error squares each miss first, so one big miss dominates. Root mean squared error puts the squared version back into the units of the target. Worked on four misses of 1, 1, 1 and 5:

Metric	Working	Result	Units
MAE	(1 + 1 + 1 + 5)/4	2.00	same as $y$
MSE	(1 + 1 + 1 + 25)/4	7	$y^2$
RMSE	√7	2.65	same as $y$

RMSE is never below MAE, and they are equal only when every miss is the same size. A big gap between them is a sign of a few large misses. Choose by the cost shape: if a miss twice as large costs twice as much, use MAE; if it costs far more than twice, use RMSE. Squared error is also the one that chases outliers when fitting, which is why a crash day can tilt a whole model.

Real run. Next month’s volatility on the 2,405 test months, a linear model against the baseline of always predicting the training mean:

Model	MAE	MSE	RMSE	Largest miss
Predict the training mean	8.19	96.8	9.84
Linear regression	5.85	56.3	$\key{7.50}$	30.9

The RMSE sits well above the MAE for the linear model, because a handful of months (March 2020 is one) were missed by thirty points.

Compared with what: the baseline

The first panel of the chart has a dotted horizontal line at 22.5, the training mean. Every model has to beat that line, and in the ten-ideas post a fully grown tree did not: it scored 15.69 against the baseline’s 9.84, having memorised the training months. A number like 15.69 looks fine in isolation and is worse than doing nothing.

Problem	Zero-skill baseline
Regression	predict the training mean every time
Classification	predict the majority class every time
A trading rule	buy and hold the index over the same period

For the stress-month problem, 77.4% of test months are calm, so “always say calm” scores 77.4% accuracy. A model reporting 78% accuracy has learned almost nothing.

Classification: the confusion matrix

A classifier first produces a probability, then a threshold turns it into a yes or no. The four outcomes:

Item	Predicted stress	Predicted calm
Actually stress	true positive (TP)	false negative (FN), $\bad{\text{a missed stress month}}$
Actually calm	false positive (FP), a false alarm	true negative (TN)

Four rates come out of it, and the way to keep them straight is what they divide by:

Rate	Formula	Divides by	Answers
Accuracy	(TP + TN) / all	everything	how often right, overall
Recall (sensitivity)	TP / (TP + FN)	the actual stress months	of the real stress months, how many did we catch?
Specificity	TN / (TN + FP)	the actual calm months	of the calm months, how many did we leave alone?
Precision	TP / (TP + FP)	the predicted stress months	when we say stress, how often are we right?

Recall and specificity divide by the true counts; precision divides by the predicted count. The false alarm rate is one minus specificity.

Real run. A random forest on the test months, 544 of which were stress months. Same probabilities, three thresholds:

Threshold	TP	FP	FN	TN	Accuracy	Recall	Specificity	Precision
0.5	150	124	394	1,737	78.5%	$\bad{27.6\%}$	93.3%	54.7%
0.3	302	275	242	1,586	78.5%	$\key{55.5\%}$	85.2%	52.3%
0.2	422	489	122	1,372	74.6%	77.6%	73.7%	46.3%

Accuracy says the first two rows are identical. They are not: at 0.5 the model misses 394 of the 544 stress months; at 0.3 it misses 242 and raises 151 more false alarms. Which row is right depends on what a miss costs. If a missed stress month means an unhedged loss and a false alarm means a few days of unnecessary hedging, the lower threshold is the better model, and accuracy would never have told you. Move the threshold to match the costs, and report the rate that matches the question.

ROC and AUC: every threshold at once

Slide the threshold from 1 down to 0 and plot recall against the false alarm rate at each step. That curve is the ROC curve, the third panel of the chart. A model with no skill runs along the diagonal; a perfect one goes straight up the left edge. The area under it, AUC, summarises the curve: 0.5 is a coin flip, 1.0 is perfect. The forest scores $\key{0.821}$ . AUC is threshold-free, which makes it the right single number for comparing models; it is the wrong number for deciding what to do, because doing something needs a threshold.

The fair-comparison checklist

Two scores are comparable only if they share all of these:

the same rows, over the same period, with the test later than the training;
the same metric in the same units (a percentage point is 0.01 in decimal returns);
preprocessing (scaling, filling, feature selection) fitted on the training part only;
several runs, with the spread reported, not the best run.

One run is one draw. Stable across random seeds is not the same as robust to a change of regime.

Traps

Trap	Truth
Tuning on the test set is fine when data is scarce	It turns the test set into training data
MAE punishes outliers more than RMSE	The reverse
MSE is in the target’s units	Squared units; RMSE is in the target’s units
MAE can exceed RMSE	Never
78% accuracy means a good model	Not when 77% of rows are the majority class
Higher directional accuracy means more profit	Sizes and costs sit in between
The same AUC means the same decisions	AUC is threshold-free; decisions need a threshold

The code

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix, roc_auc_score, roc_curve

e = y_test - y_pred
mae, mse, rmse = np.abs(e).mean(), (e ** 2).mean(), np.sqrt((e ** 2).mean())
baseline = np.sqrt(((y_test - y_train.mean()) ** 2).mean())        # the number to beat

clf = RandomForestClassifier(300, min_samples_leaf=20, random_state=0).fit(X_train, stress_train)
prob = clf.predict_proba(X_test)[:, 1]                             # a probability per month
for threshold in (0.5, 0.3, 0.2):
    tn, fp, fn, tp = confusion_matrix(stress_test, prob >= threshold).ravel()
    recall, precision = tp / (tp + fn), tp / (tp + fp)
auc = roc_auc_score(stress_test, prob)                             # 0.821
fpr, tpr, thresholds = roc_curve(stress_test, prob)                # the ROC curve's points

Summary

Idea	Rule	Remember it as
Three roles	train, validate, test once	the test set is spent on first sight
MAE, MSE, RMSE	mean miss; mean squared miss; its root	RMSE ≥ MAE; a gap means big misses
Baseline	the mean, the majority class, buy and hold	what zero skill scores
Recall, specificity	divide by the true counts	of the real cases, how many caught
Precision	divide by the predicted count	when we say yes, how often right
Threshold	moves recall against false alarms	set it by the cost of each mistake
AUC	area under the ROC curve	compare models, then still choose a threshold

Bagging: many copies of an unstable model, averaged, and what that does to the test score.

References

James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), sections 2.2 and 4.4.2: assessing accuracy, the confusion matrix, thresholds and ROC
scikit-learn: metrics and scoring, the definitions behind every function used here
scikit-learn: common pitfalls and recommended practices, on data leakage into test scores