Time series: returns are the data, prices are not

A stock’s price history looks like any other column of numbers, and that is the trap. The rows are ordered in time, and the order carries information: what came before, what was known when, and which stretches of the past still describe the present. This post works through what that order does to the three things you compute first, a correlation, an average and a volatility, on SPY’s daily prices from 2019 to 2023 (adjusted for dividends and splits, the same data the rest of the series uses). The first series showed that returns multiply; this post is about why returns, and not prices, are the thing you analyse.

Prices remember, returns forget

A daily return is the price today over the price yesterday, minus one:

r, the observed daily return

Autocorrelation is the correlation of a series with itself shifted by k days (ISLP section 10.5 defines it this way for a trading-volume series): take every pair of values k days apart and compute their ordinary correlation.

ρ, the autocorrelation being measuredr, the observed daily return

For SPY’s price the number at lag 1 is $\key{0.998}$, and at lag 20 it is still $\key{0.962}$. Today’s price is almost exactly yesterday’s price plus a small step, so a price “predicts” the next price perfectly and tells you nothing about the step, which is the only part you can trade. For the daily return the lag-1 autocorrelation is $\key{-0.167}$, and at most lags it sits near zero. That is the first habit: returns are the data. A price level also says nothing about risk. A five-dollar move is 20% of a 25-dollar stock and 1% of a 500-dollar one, and a return is the same number whichever stock it came from.

What “near zero” means, measured honestly

How small does an autocorrelation have to be before it is just luck? Instead of a formula, ask the data. Shuffle the 1,257 daily returns into a random order, which destroys any real dependence between neighbours, and compute the lag-1 autocorrelation of the shuffled series. Do that a thousand times. In 95% of the shuffles the number stayed below $\key{0.054}$ in size. Anything inside that band is what no information looks like.

By that test the $\key{-0.167}$ at lag 1 is real. It is also one year’s doing. Measured on 2020 alone the lag-1 autocorrelation is $\key{-0.319}$: in the crash and the rebound, a big down day was often followed by a big up day. Measured on the other four years together it is $\key{-0.010}$, inside the luck band. A five-year statistic was set by three months. This is what non-stationarity means in practice (a series whose behaviour changes over time): a number you compute over a long sample can describe one regime and none of the others, and the first question to ask of any historical estimate is which stretch of history produced it.

A statistic has a date range. Before trusting a correlation, an average or a volatility computed on years of data, split the years and see whether the number survives.

The size of a move clusters, even when its direction does not

Drop the sign of each daily return and keep its size. Now the lag-1 autocorrelation is $\key{0.381}$, and it is still $\key{0.197}$ twenty days out, far outside the luck band of about 0.05. Yesterday’s return does not tell you whether today goes up or down, but it does tell you how far today is likely to move. Without 2020 the lag-1 figure for sizes is $\key{0.146}$, smaller but still real, so unlike the signed figure this one is not a single year’s artefact.

The chart’s second panel shows the same thing by eye. The 63 days in the top 5% of move sizes do not arrive evenly: 30 of them fall in three months (March and April 2020, October 2022), and 42 of the 60 months contain none. The day after a top-5% day, SPY moved $\key{2.10\%}$ on average in either direction; after any other day, $\key{0.81\%}$.

This is volatility clustering, and it is why a single volatility number for a whole sample misleads. Over the five years SPY’s volatility is $\key{21.0\%}$ a year. Measured over trailing 20-day windows it ranged from $\key{4.8\%}$ (November 2019) to $\key{93.2\%}$ (late March 2020). Risk is far more predictable than return, which is why position sizes that scale with recent volatility are so common: the question “how big is tomorrow’s move likely to be” has a usable answer, and the question “which way” mostly does not.

A window that ends today, never one that straddles it

A rolling statistic is a statistic computed over a window that slides along the series: a 20-day average, a 20-day volatility, a 60-day correlation (Hilpisch, chapter 8, “Rolling Statistics”). The window has a length and a position, and the position is the trap. A trailing window ends on the day it is stamped: the average for 23 March 2020 uses 4 March to 23 March. A centred window puts the stamped day in the middle: the average for 23 March uses 9 March to 6 April, ten trading days that had not happened yet.

The third panel shows what that buys. SPY’s 2020 low was 23 March. The centred average turned up on 26 March, three days later, because it already contained the April rebound. The trailing average turned up on 8 April, nine trading days after that. A rule built on the centred average would have looked clairvoyant in a backtest and would be unbuildable live, since on 23 March the data to compute it did not exist. Hilpisch’s footnote on the same page calls this foresight bias: using, at some point in the analysis, data that only becomes available later.

The same logic rules out shuffled train-test splits for dated rows. Split by time, train on the earlier years and test on the later ones, so the test never contains a day the model’s training data came after. Every feature has a timestamp works through the feature-level version of this.

Information fades with distance

Look again at the first panel. The size-of-move autocorrelation is $\key{0.381}$ one day out and $\key{0.197}$ twenty days out. Whatever yesterday tells you about today, it tells you less about next month, and that holds for almost every signal built from prices. A forecast has a horizon it is good for, and the holding period of a trade has to match it: a signal that fades in a day is worth little to a strategy that trades monthly, and a monthly signal is noise to a strategy that trades daily. Later posts return to this when they ask how fast an alpha decays.

The code

The snippet reproduces the lag-1 numbers and the luck band. Variables:

import numpy as np
import pandas as pd

url = "https://raw.githubusercontent.com/ankurtrapasiya/examples/main/data/prices.csv"
px = pd.read_csv(url, index_col="date", parse_dates=True)["SPY"]
r = (px / px.shift(1) - 1).dropna()          # today's price over yesterday's, minus one

print("price,  lag 1:", round(px.autocorr(1), 3))
print("return, lag 1:", round(r.autocorr(1), 3))
print("size,   lag 1:", round(r.abs().autocorr(1), 3))
print("return, lag 1, without 2020:", round(r[r.index.year != 2020].autocorr(1), 3))

rng = np.random.default_rng(0)
shuffles = [pd.Series(rng.permutation(r.values)).autocorr(1) for _ in range(1000)]
band = np.quantile(np.abs(shuffles), 0.95)   # order destroyed: how big can it get by luck?
print("luck band:", round(band, 3))

It prints 0.998 for the price, −0.167 for the return, 0.381 for the size, −0.010 without 2020, and a luck band of 0.054.

Summary

References

• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning with Applications in Python (Springer, 2023). Section 10.5 defines autocorrelation at lag ℓ as the correlation of all pairs of values ℓ days apart, on the NYSE trading-volume series.
• Yves Hilpisch, Python for Finance, 2nd edition (O’Reilly, 2018). Chapter 8, “Rolling Statistics”, for rolling windows in pandas, and its footnote on foresight bias.
• Yuxing Yan, Python for Finance, 2nd edition (Packt, 2017). Chapter 8 for the daily log and simple return definitions and the fat tails of S&P 500 daily returns.
• Romero and Balch, What Hedge Funds Really Do (Business Expert Press, 2014). Chapter 12 for why prices are adjusted for splits and dividends before any of this is computed.