The data is part of the model: splits, gaps and outliers

Before a model learns anything, someone has already made three choices about its data, usually without noticing: which rows it trains on and which it is tested on, what to do when a value is missing, and how far one extreme value is allowed to pull the result. Each of these choices can move the score as much as swapping in a cleverer model. This post makes each one on purpose and measures what it changes, on the series’ problem: predict next month’s volatility (how much a price will swing) from this month’s prices.

First, two tools the rest of the post leans on. Both get their own posts later in the series; this is all you need for now.

a chosen way of scoring or fillingerrorthe outlier

Two tools this post uses

A fitted line

Put each month on a chart as a dot: this month’s volatility along the bottom, next month’s up the side. Wild months tend to be followed by wild months, so the dots drift upwards. A linear regression draws the one straight line that fits them best, and you read a forecast off it: find this month’s volatility on the bottom, go up to the line, read across. How “best” is decided is the subject of the linear regression post; for now Figure 1 is enough.

The top panel uses five real SPY months. No straight line passes through every dot. The gap between a dot and the line, actual minus predicted, is the miss; statisticians call it the residual. June 2020 came in 10.1 points above the line, August 2020 only half a point below it.

One number for many misses

To compare two models you need one number that sums up all their misses. Three are common, worked in Table 1 on the five misses above (+10.1, −0.5, −5.4, −5.3 and +1.0):

(Computed from the unrounded misses; the rounded ones shown give 4.46, 32.1 and 5.67.)

Read RMSE as “the typical size of a miss”, with one twist: the misses are squared before they are averaged, so big misses count extra. The bottom panel shows how much extra. Counted as they are, June 2020’s miss is 45% of the total; squared, it is 64%, and August’s half point almost disappears. For all three scores, lower is better. The rest of this post scores models by RMSE, and the squaring comes back in the section on outliers.

The split decides what the score means

To score a model you hide some of the data from it, train on the rest, and check its guesses on the hidden part. Doing that once gives one score from one slice of the data. Cross-validation repeats it: cut the data into five parts, called folds, let each part take one turn as the hidden test set while the model trains on the other four, and average the five scores. How you cut the parts matters:

• Shuffled folds: the months are mixed up at random before cutting, so each part is a random handful of months from all six years.
• Forward folds: the months stay in time order. Each round trains only on the past and tests on the block that comes next, the way the model would be used for real. This is also called walk-forward testing.

The score is RMSE from the section above: the typical size of a miss, in points of volatility. The same random forest (a model that averages many decision trees, explained later in the series), scored three ways, next to the baseline (Table 2):

The three numbers are not scored on the same months, so they are not three opinions about one question. The shuffled folds score best, and that is the leak: every test month had its neighbouring months in training, so the forest gets credit for recognising them. They are also the only scoring that tests on 2020, the crash year. The forward folds train on 2020 and never test on it: their first test block starts in January 2021. Their five scores were 13.30, 13.37, 11.98, 10.43 and 14.08. The first fold, trained on 2020 alone, the crash year, was then asked about 2021, a calmer one, and missed by thirteen points; the fourth, testing on late 2023 and 2024, missed by ten; the last, testing on 2025, by fourteen. The single time split, trained on all four years, gives one number for 2024 to 2025. One averaged number hides how much the miss moves from one kind of market to the next.

For a time series, the test must be later than the training. Each row’s volatility is measured over a window of about a month, so two rows a day apart are built from almost the same days: near-twins. A random split puts a near-twin of every test row into the training set, and the model gets credit for recognising it, which it could never do live. Forward folds are the version of cross-validation that respects time; the second panel of Figure 2 draws the five used here. The scikit-learn post showed the same effect on a yes/no model, scored by AUC (0.5 is a coin flip, 1 is perfect): shuffled folds promised 0.929, forward folds 0.899.

A second rule follows. Every step that learns from the data is part of training: the scaler (which puts features on the same scale, using their average and spread), the imputer (which fills gaps, often with an average), the choice of which features to keep, and the choice of settings. Each of them learns numbers from the rows it sees. Let them see only the training part of each fold, never the whole table, or the test rows leak into training.

Gaps: why a value is missing matters more than that it is

A value can be missing for three kinds of reason, and the reason decides what you may do (Table 3):

Real run, the third kind. I removed the 20-day volatility from 12.6% of the training rows, with the removal more likely when the volatility was high, the way a stock goes missing in real data when trading in it is stopped or thin. Then three repairs, scored in Table 4:

The three repairs land within a quarter of a point of each other, and dropping the rows even came in 0.13 below the run with nothing missing. Differences that small do not rank the repairs: on this window the gaps cost nothing the score can see, and the flag bought 0.03 over plain filling. The flag is still the right habit, because it costs nothing and is the only repair that keeps the fact of the gap. For prices the standing rule is different and simpler: fill forward, so a day with no trading repeats the last known price, and fill backward only where nothing came before.

Outliers: the loss function decides how far they pull

The line in Figure 1 is the one whose misses add up to the smallest total. The loss is the rule for counting each miss in that total. Ordinary linear regression squares them, exactly like the MSE, and that choice decides how hard one far-out point can pull. Three misses of 1, 1 and 10 (Table 5):

Squaring is why one big miss dominates, the same effect as June 2020 in Figure 1. Real run. Forty points around a line of slope 0.5, then the last point moved about 23 units above the line (Table 6):

One point in forty moved the least-squares slope by two thirds. The third panel of Figure 2 draws it. Where the point sits matters: this one is at the edge of the range (x = 10), where it can tilt the line. The same miss in the middle (x = 5) leaves the slope at 0.51 and only lifts the line. A point at the edge like this is called a high-leverage point. A crash day can be exactly that, an extreme value at the edge of everything the model has seen, and squared-error models, which are most of them, feel it most. The choices are to use a robust loss, to use a model where only the side of a split matters (a tree), or to keep the squared loss and know what it is doing. Deleting the point is not on the list: a crash is real data, and a model that has never seen one will meet one live.

The rules

1. Split by time; test later than training; walk forward for cross-validation.
2. Fit scalers, imputers and feature selection inside the training part of each fold.
3. Ask why each value is missing before deciding what to do about it; keep a missingness flag.
4. Know your loss; squared error chases outliers.
5. Never delete a crash.

Traps

Things people say about splits, gaps and outliers, and what is true (Table 7):

The code

All three ideas on made-up data, so it runs as pasted (pip install numpy scikit-learn). Part 1 builds a daily series whose volatility drifts slowly, like real volatility, and scores one random forest with shuffled and with forward folds. X holds the one feature, last 20 days’ volatility, one row per day; y holds the answer for each row, the next 20 days’ volatility.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import HuberRegressor, LinearRegression, QuantileRegressor
from sklearn.model_selection import KFold, TimeSeriesSplit, cross_val_score

# 1. Shuffled folds against forward folds --------------------------------------
# Made-up daily returns whose volatility drifts slowly, as real volatility does.
rng = np.random.default_rng(0)
n = 1500                                                  # about six years of trading days
vol = np.exp(np.cumsum(rng.normal(0, 0.05, n)))           # daily volatility, wandering over time
returns = rng.normal(0, vol)                              # one return per day
past_20d = np.array([returns[t - 20:t].std() for t in range(20, n - 20)])
next_20d = np.array([returns[t:t + 20].std() for t in range(20, n - 20)])

X = past_20d.reshape(-1, 1)   # one feature: volatility over the last 20 days
y = next_20d                  # the answer: volatility over the next 20 days
# Rows a day apart share 19 of their 20 days, so neighbouring rows are near-twins.

rf = RandomForestRegressor(200, min_samples_leaf=5, random_state=0)
for name, cv in [("shuffled folds", KFold(5, shuffle=True, random_state=0)),
                 ("forward folds ", TimeSeriesSplit(5))]:
    scores = -cross_val_score(rf, X, y, cv=cv, scoring="neg_root_mean_squared_error")
    print(f"{name}: RMSE per fold {np.round(scores, 3)}, average {scores.mean():.3f}")

# 2. A missing value: fill it, and keep the fact that it was missing ------------
x = np.array([12.0, np.nan, 15.0, np.nan, 11.0])          # a feature with two gaps
was_missing = np.isnan(x)                                 # [False True False True False]
x_filled = np.where(was_missing, np.nanmean(x), x)        # nanmean: the average, ignoring gaps
print(np.column_stack([x_filled, was_missing]))           # two columns go to the model

# 3. One outlier against three losses --------------------------------------------
rng = np.random.default_rng(2)
xs = np.linspace(0, 10, 40)
ys = 2 + 0.5 * xs + rng.normal(0, 0.5, 40)                # 40 points around a line of slope 0.5
ys_out = ys.copy()
ys_out[-1] = 30.0                                         # move the last point far above the line
for name, model in [("squared (least squares)", LinearRegression()),
                    ("absolute (median line) ", QuantileRegressor(quantile=0.5, alpha=0, solver="highs")),
                    ("Huber                  ", HuberRegressor())]:
    clean = model.fit(xs.reshape(-1, 1), ys).coef_[0]
    dirty = model.fit(xs.reshape(-1, 1), ys_out).coef_[0]
    print(f"{name}: slope {clean:.2f} clean, {dirty:.2f} with the outlier")

It prints, in order:

• Folds: shuffled folds average about 0.14, forward folds about 0.20, with one forward round much worse than the rest. That is the same pattern as Table 2, on different numbers.
• Missing value: the filled column (the gaps became the average, 12.67) next to the “was missing” column (1 where there was a gap).
• Outlier: the slopes from Table 6 exactly: 0.50 and 0.83, 0.51 and 0.56, 0.51 and 0.53.

The script behind the real numbers

The tables above come from examples/data-is-part-of-the-model/sampling.py, which runs the same steps on the blog’s five tickers; the two-tools chart comes from examples/data-is-part-of-the-model/two_tools.py, which needs only the five months typed into it. To run it yourself, clone the examples repo: its README has a three-command Docker setup, or run uv run on it.

Summary

Table 8 is the post in five rows.

References

• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 2.2.1: the mean squared error; section 3.1.1: the least-squares line; section 3.3.3: outliers and high-leverage points; section 5.1.3: k-fold cross-validation
• Stefan Jansen, Hands-On Machine Learning for Algorithmic Trading (Packt, 2018), chapter 6, “Challenges with cross-validation in finance”: why shuffled folds fail on market data, and walk-forward testing
• scikit-learn: cross-validation of time series data, and common pitfalls
• Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), section 1.5.5: the three kinds of missing data (MCAR, MAR, NMAR); section 11.6: robust regression, including the Huber loss (11.6.3)