Every feature has a timestamp: leakage, scaling and twin features

A model is only as honest as its inputs. This post builds a dishonest input on purpose, shows how it looks in the score and how to catch it, and then covers the two quieter feature problems: units and twins.

a coefficient or a chosen penaltyerrorthe leak

The leak

The series’ seven features are all computed from a ticker’s prices up to the day of the forecast: recent returns, recent volatility, distance from the moving average. I added an eighth on purpose: SPY’s volatility over the same month the model is trying to predict. It is a perfectly reasonable-looking column, the kind that appears when a market-wide table is joined onto a per-ticker table by month without thinking about which month. The score is the test RMSE, the typical size of a miss in points of volatility; lower is better (Table 1).

The leak looks like the best model in Table 1. The honest version of the same idea, last month’s market volatility, adds a third of a point. Nothing in the score distinguishes the two; the only difference is a timestamp.

How to catch it. Lag the suspicious feature by one period and rerun. The leaked column shifted back by twenty trading days scores 12.40 (the third bar in Figure 1), next to the honest feature’s 12.48 and nowhere near 11.19: most of the “signal” vanished, which means it was never a signal about the future, only a copy of it. Removing the feature is the same test. Run either on any result that looks too good.

Leakage comes in two shapes. Time leakage uses the future: a same-month statistic, a revised figure, a label that overlaps the test period. Pooling leakage uses the test rows: a mean, a scaler or a feature-selection step computed over the whole table. A penalty on the model’s weights (ridge and lasso, below) does not stop either, and a correct split cannot repair a feature that was built looking forward.

Scaling: units, not information

Some models measure distances between rows (nearest neighbours), or penalise the size of their coefficients, the weights a linear model gives each feature (ridge, lasso, logistic regression). Both depend on the units of each feature, so features are standardised: each column rescaled to a mean of zero and a spread of one. Scaling changes a feature’s influence, not its information. Trees do not need it, because a split only asks whether a value is above or below a threshold.

The rule that matters is where the scaler is fitted. Fit it on the training rows and apply it to the test rows. Fitted on all rows together, it carries the test set’s mean into the model: in this data the pooled mean of 20-day volatility is 28.53 points and the training-only mean is 29.94, because the pooled version already knows 2024 and 2025 were calmer than 2020 to 2023. The difference is small here; the habit is the point, and scikit-learn’s Pipeline exists so that the habit is automatic.

Twin features: ridge shares, lasso chooses

Add a column that is 1.02 times the 20-day volatility plus a hair of noise, a twin of a feature already there. Then fit three models. Least squares is the plain linear model. Ridge and lasso are the same model with a penalty added to the total it minimises, a cost for having large weights, so the fit keeps the weights small unless they earn their place. The penalty’s strength, written $\lambda$ (lambda), is a setting you choose (Table 2):

Least squares gives the twins wildly opposite weights that cancel; the predictions are fine, the coefficients are meaningless, and a tiny change in the data would swap them. Ridge pulls both a long way towards zero; in this data they keep opposite signs, at a fraction of the size. Lasso drops one exactly. The sum is stable in every row; the split between the twins is not. Correlated features break the interpretation of a linear model before they break its predictions, and in a random forest they split the importance score between them without breaking anything.

The two penalties side by side (Table 3):

The second panel of Figure 1 draws both paths on the seven honest features as the penalty grows: lasso’s lines hit zero one by one (six features left at a penalty of 0.1, three at 1), ridge’s lines approach zero and never arrive. The penalty is chosen on validation data, rows held back from training for picking settings like this. At zero it is plain least squares; large enough, it predicts the mean.

A rolling z-score hides a crash

A common feature is a rolling z-score: how far today’s price sits from its recent average, in recent standard deviations. It has a flaw that shows up exactly when it matters (Table 4).

The same six-point gap reads as extreme in a calm window and as nothing in a wild one, because the wild window’s standard deviation has ballooned. SPY in the spring of 2025: a week into the fall, on 27 February, the 60-day z-score read −1.66. At the bottom on 8 April, with the price 83 points below the window’s mean, it read −3.20, but only because the window’s standard deviation had tripled from 8.1 to 25.8 along the way; measured against the calm February window it would have read near −10. The third panel of Figure 1 shows the z-score dropping past −3 at the bottom of the worst fall of the window and then snapping back above zero within weeks. A feature built to flag extremes undersold the biggest one, and said nothing about how fast it would end.

Traps

Things people say about features, and what is true (Table 5):

The code

All four ideas on made-up data, so it runs as pasted (pip install numpy scikit-learn); scaling is the StandardScaler step in the twins part. It builds a daily series whose volatility drifts slowly, like real volatility. past_20d is the honest feature (the last 20 days’ volatility, knowable today) and y is the answer (the next 20 days’ volatility). The first 800 rows train the model and the rest test it, in time order.

import numpy as np
from sklearn.linear_model import Lasso, LinearRegression, Ridge
from sklearn.metrics import root_mean_squared_error
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Made-up daily returns whose volatility drifts slowly, as real volatility does
rng = np.random.default_rng(0)
n = 1200
vol = np.exp(np.cumsum(rng.normal(0, 0.05, n)))
returns = rng.normal(0, vol)
days = range(40, n - 20)
past_20d = np.array([returns[t - 20:t].std() for t in days])   # knowable today
next_20d = np.array([returns[t:t + 20].std() for t in days])   # the answer: not knowable yet
y = next_20d
train, test = slice(0, 800), slice(800, None)                  # test later than training

# 1. The leak: a column computed over the same days as the answer
leak = next_20d + rng.normal(0, 0.02, len(y))                  # a copy of the future, with noise
leak_lagged = np.r_[np.full(20, leak[0]), leak[:-20]]          # the same column, 20 days older
for name, X in [("honest feature only       ", np.c_[past_20d]),
                ("plus the leak             ", np.c_[past_20d, leak]),
                ("plus the leak, lagged 20 d", np.c_[past_20d, leak_lagged])]:
    model = LinearRegression().fit(X[train], y[train])
    print(name, "test RMSE", round(root_mean_squared_error(y[test], model.predict(X[test])), 3))

# 2. Twin features: a second column that says almost the same thing as the first
twin = 1.02 * past_20d + rng.normal(0, 0.001, len(y))
X = np.c_[past_20d, twin]
for name, model in [("least squares", LinearRegression()),
                    ("ridge        ", Ridge(alpha=1.0)),
                    ("lasso        ", Lasso(alpha=0.01))]:
    pipe = make_pipeline(StandardScaler(), model).fit(X[train], y[train])  # scaler sees training rows only
    w = pipe[-1].coef_                                         # the weights, one per column
    print(name, "weights", np.round(w, 3), "sum", round(w.sum(), 3))

# 3. A rolling z-score: the same 6-point gap, in a calm window and in a wild one
for mean, sd, price in [(100, 2, 106), (100, 10, 106)]:
    print(f"window mean {mean}, sd {sd}, price {price}: z = {(price - mean) / sd:.1f}")

It prints:

• The leak: the leaked column cuts the test miss from about 0.083 to 0.020, a model that looks far better. Lagged by 20 days it scores 0.084, no better than without it: it was a copy of the answer, not a signal.
• Twins: least squares gives the twins opposite weights (about −0.81 and +1.18), ridge splits the weight evenly (0.18 and 0.185), lasso keeps one and sets the other to exactly 0. The sum stays about the same in every row, as in Table 2.
• z-scores: 3.0 in the calm window, 0.6 in the wild one, the two rows of Table 4.

np.c_[a, b] puts columns side by side into a table, and pipe[-1] is the last step of the pipeline, the model itself.

Summary

Table 6 is the post in six rows.

References

• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 3.3.3: collinearity; section 6.2: ridge and the lasso, and when each does better (6.2.2, “Comparing the Lasso and Ridge Regression”)
• scikit-learn: common pitfalls and recommended practices, the section on data leakage
• Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), section 11.7.6: why twin features get unstable weights with a stable sum; section 11.4.8: ridge’s “grouping effect” on highly correlated features