Data that lies: the bugs that look like alpha

A backtest is a claim about what would have happened. The claim is only as good as the data it ran on, and financial data fails in a handful of well-known ways. This post is the catalogue (Table 1): each bug, what it does to a result, how to catch it, and the fix. Three of them are demonstrated on purpose.

a chosen rule or estimatea gain that looks realwhat it was worth once the bug is removed

The catalogue

Data snooping: a thousand ideas that know nothing

Make 1,000 trading signals by flipping a coin every day: long SPY on heads, short on tails. Score each on 2020 to 2023 with a t-statistic: how many standard errors its average daily result sits above zero, where above 2 is the usual bar for “significant”. By luck alone, about 2.3% of them should clear it (Table 2).

The twenty-three “discoveries” had an average Sharpe of 1.24 (return per unit of risk), a number a fund would advertise, and every one of them was a coin. Twenty-three is also exactly the number luck predicts. Afterwards they averaged +0.34, and the best of them kept a Sharpe of 1.59: the same coins got lucky twice, in a window where SPY rose most months. The first panel of Figure 1 is the whole lesson: a cloud with no slope, a correlation of 0.06 across all 1,000. What these signals scored where they were chosen says nothing about what they scored next.

The fix is to count. Testing $N$ ideas at a 5% bar expects $0.05 N$ false positives; with 1,000 ideas that is 50. The Bonferroni correction divides the bar by the number of tests: 0.05/1,000 is 0.00005, a $t$ of about 3.9. Harvey, Liu and Zhu’s rule of thumb for new factors is $t > 3$ rather than 2, because of how many have been tried. And if test results guided the design, the holdout has become training data, however carefully it was kept apart on disk.

Look-ahead: one missing shift

The backtesting post showed the simplest form: a crossover rule that sets today’s position from today’s close reports +113.6% on SPY; the same rule set from yesterday’s close reports +106.7%. Seven points from a missing .shift(1).

The subtler form hides in preprocessing. Build a “high volatility” flag as a z-score of SPY’s 20-day volatility, centred on the full 2020 to 2025 mean. The flag fires on 116 days. Build it with only the mean known so far, and it fires on 59 different-looking days; the two first disagree on 4 February 2020, because the full-sample version already knows that March 2020 is coming and sets its bar accordingly. The third panel of Figure 1 draws both. Any statistic computed over train and test together, a mean, a standard deviation, a feature-selection step, a scaler, leaks the future into the past. Fit every data-dependent step on the training rows only.

The test for a leak: shift the suspect feature by a day, or remove it, and rerun. If earlier predictions change, something was looking forward. A result that is too good is a result to check before it is a result to report.

Survivorship: built on purpose

A survivor-only universe is what you get by downloading today’s index members and testing on their past. To see what it does, I made 200 stocks from scratch: three years of random daily returns with a small common drift, and a slightly different drift per stock. At the end, any stock that had lost more than half its value was dropped from “the index”. 25 of the 200 were (Table 3).

The survivors return more, by construction: the ones that fell were removed. The damage is worst for strategies that buy what has fallen, because in a survivor-only universe the stocks that fell and stayed down are missing, so the fallers left all look like recoveries. Chan (chapter 3) gives a real-data example: the same buy-the-losers rule returned −42% on a full database and +388% on a survivor-only one. More survivor data does not fix this; it is a systematic bias, not a sampling error.

Adjusted prices, dividends and the dead

A raw price series misleads in three ways (Table 4):

The adjusted close folds splits and dividends into earlier prices: a 2-for-1 split halves every earlier price, and a dividend trims them by its yield, so the day-to-day returns are what a holder actually earned. Fill price gaps forward first, then backward only where nothing came before, so a trading halt repeats the last known price instead of inventing one. Keep the dead tickers in the price folder; a bankrupt company’s symbol often ends in Q, and a universe that still has those files is what point-in-time looks like on disk.

The fixes, in order of effort

1. Lag everything. Every feature uses data up to the previous close. One line.
2. Fit preprocessing on training rows. Scalers, imputers, feature selection, inside the pipeline, inside each fold.
3. Count the trials, and raise the bar to match. Keep a research log of what was tried, including what failed.
4. Hold out time, not rows, and touch the holdout once.
5. Use adjusted, total-return, point-in-time data, delistings included.
6. Shift and ablate. Lag the suspect, remove the suspect, rerun. If the result moves more than it should, it was a leak.

Traps

Table 5 is the list of mistakes, with the fix beside each.

The code

The thousand coins and the leaky flag. It reads the blog’s price file from the examples repo, so it runs as pasted (pip install numpy pandas). tr and te hold SPY’s daily returns for 2020 to 2023 and for 2024 to 2025; sig_tr holds the coin flips, one row per signal and one column per day; pnl_tr each signal’s daily result.

import numpy as np
import pandas as pd

URL = "https://raw.githubusercontent.com/ankurtrapasiya/examples/main/data/prices.csv"
prices = pd.read_csv(URL, index_col="date", parse_dates=True)
r = prices.pct_change().dropna()                 # daily returns
tr = r.loc[:"2023", "SPY"].to_numpy()            # the years the signals are chosen on
te = r.loc["2024":, "SPY"].to_numpy()            # the years after

# 1,000 signals that are pure coin flips: +1 = long SPY that day, -1 = short
rng = np.random.default_rng(1)
sig_tr = rng.choice([-1.0, 1.0], size=(1000, len(tr)))   # one row per signal, one column per day
sig_te = rng.choice([-1.0, 1.0], size=(1000, len(te)))   # the same coins, flipped again later
pnl_tr, pnl_te = sig_tr * tr, sig_te * te                # each signal's daily result

def sharpe(pnl):                                         # yearly Sharpe ratio of each row
    return np.sqrt(252) * pnl.mean(1) / pnl.std(1, ddof=1)

t = pnl_tr.mean(1) / pnl_tr.std(1, ddof=1) * np.sqrt(len(tr))   # a t-statistic per signal
lucky = t > 2                                                    # "significant" by the usual bar
print(lucky.sum(), "of 1,000 look significant")                  # 23
print(round(sharpe(pnl_tr)[lucky].mean(), 2), round(sharpe(pnl_te)[lucky].mean(), 2))   # 1.24 0.34
print(round(np.corrcoef(sharpe(pnl_tr), sharpe(pnl_te))[0, 1], 2))                      # 0.06

# Look-ahead in preprocessing: a high-volatility flag built two ways
vol = r["SPY"].rolling(20).std()                                 # 20-day volatility
leaky = (vol - vol.mean()) / vol.std()                           # uses the whole 2020 to 2025 window
honest = (vol - vol.expanding().mean()) / vol.expanding().std()  # uses only what was known so far
print((leaky > 1).sum(), (honest > 1).sum())                     # 116 59: different days flagged

It prints Table 2: 23 of 1,000 significant, their Sharpe of 1.24 on the years they were picked and +0.34 afterwards, a correlation of +0.06 between the two, and the flag firing on 116 days with the full-sample mean against 59 with only past data. .mean(1) averages along each row (one signal’s days); .expanding() uses every day up to the current one and none after.

Summary

Table 6 collects the rules.

References

• Campbell R. Harvey, Yan Liu and Heqing Zhu, ”… and the Cross-Section of Expected Returns”, The Review of Financial Studies 29(1), 5–68, 2016, the case for a t-statistic above 3
• scikit-learn: common pitfalls and recommended practices, data leakage through preprocessing
• Ernest P. Chan, Quantitative Trading: How to Build Your Own Algorithmic Trading Business (Wiley, 2009), chapter 3: survivorship bias (with the −42% against +388% example), look-ahead bias and adjusting for dividends
• Philip J. Romero and Tucker Balch, What Hedge Funds Really Do: An Introduction to Portfolio Management (Business Expert Press, 2014), chapter 12: adjusted prices, missing data and missing tickers