Every feature has a timestamp: leakage, scaling and twin features

A model is only as honest as its inputs. This post builds a dishonest input on purpose, shows how it looks in the score and how to catch it, and then covers the two quieter feature problems: units and twins.

a coefficient or a chosen penaltyerrorthe leak

The leak

The series’ seven features are all computed from a ticker’s prices up to the day of the forecast: recent returns, recent volatility, distance from the moving average. I added an eighth on purpose: SPY’s volatility over the same month the model is trying to predict. It is a perfectly reasonable-looking column, the kind that appears when a market-wide table is joined onto a per-ticker table by month without thinking about which month.

Features	Test RMSE
Seven honest features	7.50
Plus SPY’s volatility over the same month (not knowable yet)	$\bad{6.81}$
Plus SPY’s volatility over the last month (knowable)	7.56

The leak looks like the best model in the table. The honest version of the same idea, last month’s market volatility, adds nothing. Nothing in the score distinguishes the two; the only difference is a timestamp.

How to catch it. Lag the suspicious feature by one period and rerun. The leaked column shifted back by twenty trading days scores 7.55: the “signal” vanished, which means it was never a signal about the future, only a copy of it. Removing the feature is the same test. Run either on any result that looks too good.

Leakage comes in two shapes. Time leakage uses the future: a same-month statistic, a revised figure, a label that overlaps the test period. Pooling leakage uses the test rows: a mean, a scaler or a feature-selection step computed over the whole table. Regularisation does not stop either, and a correct split cannot repair a feature that was built looking forward.

Scaling: units, not information

Some models measure distances between rows (nearest neighbours), or penalise the size of coefficients (ridge, lasso, logistic regression). Both depend on the units of each feature, so features are standardised: each column rescaled to a mean of zero and a spread of one. Scaling changes a feature’s influence, not its information. Trees do not need it, because a split only asks whether a value is above or below a threshold.

The rule that matters is where the scaler is fitted. Fit it on the training rows and apply it to the test rows. Fitted on all rows together, it carries the test set’s mean into the model: in this data the pooled mean of 20-day volatility is 23.84 points and the training-only mean is 24.30, because the pooled version already knows 2022 was calmer than 2020. The difference is small here; the habit is the point, and scikit-learn’s Pipeline exists so that the habit is automatic.

Twin features: ridge shares, lasso chooses

Add a column that is 1.02 times the 20-day volatility plus a hair of noise, a twin of a feature already there. Fit three models:

Model	Coefficient on vol_20d	On its twin	Their sum	Test RMSE
Ordinary least squares	$\risk{-33.89}$	$\risk{+40.31}$	+6.41	7.50
Ridge, penalty 1	−1.37	+7.78	+6.41	7.50
Lasso, penalty 0.05	$\key{0.00}$	+6.27	+6.27	7.51

Least squares gives the twins wildly opposite weights that cancel; the predictions are fine, the coefficients are meaningless, and a tiny change in the data would swap them. Ridge pulls both towards zero and shares the weight. Lasso drops one exactly. The sum is stable in every row; the split between the twins is not. Correlated features break the interpretation of a linear model before they break its predictions, and in a random forest they split the importance score between them without breaking anything.

The two penalties:

Item	Ridge (L2)	Lasso (L1)
Adds to the loss	$\lambda \sum \key{\beta_j^2}$	$\lambda \sum \lvert \key{\beta_j} \rvert$
Effect as $\lambda$ grows	every coefficient shrinks, none reaches zero	some reach exactly zero
With twins	shares the weight	picks one
Use when	many small effects	a few real effects among many candidates

The second panel of the chart draws both paths on the seven honest features as the penalty grows: lasso’s lines hit zero one by one (seven features left at a penalty of 0.1, four at 1), ridge’s lines approach zero and never arrive. The penalty is chosen on validation data. At zero it is plain least squares; large enough, it predicts the mean.

A rolling z-score hides a crash

A common feature is a rolling z-score: how far today’s price sits from its recent average, in recent standard deviations. It has a flaw that shows up exactly when it matters.

Window mean	Window sd	Price	z
100	2	106	3.0
100	10	106	0.6

The same six-point gap reads as extreme in a calm window and as nothing in a wild one, because the wild window’s standard deviation has ballooned. SPY in 2020: a week into the fall, on 28 February, the 60-day z-score read −2.90. At the bottom on 23 March, with the price 79 points below the window’s mean, it read −2.82, because the window’s standard deviation had tripled from 8.6 to 28.1. The third panel of the chart shows the z-score barely moving through the worst month in a decade. A feature built to flag extremes went quiet in the biggest one.

Traps

Trap	Truth
Leakage is only about time	Pooling the test rows leaks too
Future returns as the label are leakage	That is the target; leakage is about the inputs
Trees need scaling	No
Correlated features break a random forest	They dilute its importance scores; they break least-squares coefficients
Lasso picks the truly important twin	It picks one, arbitrarily
Fewer features lowers bias	It raises bias and lowers variance
A z-score is capped at ±3	It is not, and in a crash it shrinks

The code

from sklearn.linear_model import Lasso, LinearRegression, Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# the leak test: lag the suspect by one period and see whether the score moves
X_lagged = X.copy(); X_lagged["suspect"] = X_lagged.groupby("ticker")["suspect"].shift(20)

# scaling inside a pipeline: the scaler is fitted on the training rows only
ridge = make_pipeline(StandardScaler(), Ridge(alpha=1.0)).fit(X_train, y_train)
lasso = make_pipeline(StandardScaler(), Lasso(alpha=0.05)).fit(X_train, y_train)
lasso[-1].coef_                                          # some entries exactly 0.0

z = (price - price.rolling(60).mean()) / price.rolling(60).std()   # the rolling z-score

Summary

Idea	Rule	Remember it as
Timestamp	when was this knowable?	lag everything to the last close
The leak test	lag or remove the suspect, rerun	a real signal survives a lag
Scaling	standardise, fitted on training rows	units, not information
Ridge	$\lambda \sum \beta^2$	shrinks all, shares twins
Lasso	$\lambda \sum \lvert \beta \rvert$	zeroes some, picks a twin
Rolling z-score	(price − mean) / sd	goes quiet in a crash

Decision trees: a model that needs no scaling, ignores the features it does not use, and finds thresholds on its own.

References

James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), sections 3.3.3 and 6.2: collinearity, ridge and the lasso
scikit-learn: common pitfalls and recommended practices, the section on data leakage
Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), section 11.7.6: why twin features get unstable weights with a stable sum