Why prices are hard to beat: efficiency, random walks and a roulette wheel

Three claims hide inside “markets are efficient”, and they are not the same claim. This post takes them apart, tests the simplest one on SPY from 2020 to 2025, and then uses a roulette wheel to show the second way a strategy can look like it works when it does not.

a measured pattern or a chosen rulea gaina loss

Three forms of efficiency

Efficiency is about what information the price already contains. The three forms, from the weakest claim to the strongest, are in Table 1:

Efficient does not mean prices are always right, or that nobody ever profits. It means no reliable edge after costs. Measure it by how fast prices react to news: big, liquid, heavily watched markets react within seconds and are the most efficient; small and obscure ones react slowly. The evidence for the semi-strong form, as Romero and Balch summarise it (chapter 8), is that very few fund managers beat their index persistently, even fewer than chance alone would produce. The people who try hardest to beat the market are the ones who make it efficient.

The random walk, tested on SPY

If past prices are useless, tomorrow’s price is today’s plus a surprise nobody can predict, and the best forecast of tomorrow’s return is zero. That is a random walk. Two things follow: yesterday’s return should tell you nothing about today’s, and streaks of up or down days should be no longer than a coin would produce.

Two of the checks in Table 2 pass: the streaks are no longer than shuffling the same days produces, and the up-after-up rate is close to the up-after-down rate. The first does not. A correlation of −0.158 at one day (the first panel of Figure 1) is a real pattern: a down day leans towards an up day next. Most of it comes from the violent weeks of March 2020, when the market reversed day after day, but it is there.

So build the rule. Hold SPY the day after a down day, sit in cash the day after an up day. With no trading costs it returned $\good{+115.4\%}$ from 2020 to 2025, against +128.4% for buy and hold (simply owning SPY throughout) and +6.9% for the opposite rule. Now charge for the 747 position changes it makes, as in the post on trading costs (Table 3):

That is what “efficient” means in practice. A pattern can be statistically real and still worth nothing, because the edge is smaller than the cost of trading it. Even at a tenth of a percent a change, the rule ends 2020 to 2025 with +2.0%, while simply holding SPY more than doubled the money. The pattern survives in the data precisely because it is too small to be worth removing.

Two patterns are tradeable in principle: momentum (what went up keeps going, from news that spreads slowly and from big orders worked in pieces) and mean reversion (what went too far comes back). Competition shortens both. A stop-loss suits momentum and hurts mean reversion, because it sells right before the bounce.

A pair trade and its z-score

The usual way to trade mean reversion is a pair: long one stock, short a related one, betting that the gap between them closes. Measure the gap as a z-score, how many standard deviations the price ratio sits from its recent average:

For GS against XOM from 2020 to 2025, the 60-day z-score of the price ratio (on a log scale, as in the code below) ran from −4.6 to +4.1 (the second panel of Figure 1) and sat beyond ±2 on 169 days. On 97% of those days the ratio was closer to its mean twenty days later. That sounds like a machine, and two things undercut it. First, “closer to the mean” is nearly guaranteed by the way the z-score is built: the mean and standard deviation move too. Second, the relationship itself drifts: the 60-day correlation between the two stocks ran from −0.15 to +0.85. A z-score measures how unusual a gap is; it is not a promise that the gap closes, and the pair can simply stop being a pair.

Arbitrage in the strict sense means identical payoffs bought and sold at the same moment: the same stock at $50.00 on one exchange and $50.40 on another, costs $0.10, is a riskless $0.30. A pair with correlation 0.8 is not that. Four limits keep real gaps from closing on schedule: costs, including carry (the cost of holding the positions while you wait); capital, and margin calls (the broker demanding more cash) while the gap widens first; crowding (everyone in the same trade exits together); and, on the short leg, the fee for borrowing the shares, or the lender recalling them. Table 4 puts numbers on four of these cases.

The roulette wheel: a high hit rate on a losing game

The second way a strategy fools you has nothing to do with costs. American roulette has 38 pockets, 18 red. A bet on red pays even money, so each spin is a bet with $p = 18/38 = 0.4737$ and expected value $\bad{-5.26}$ cents per dollar. The doubling rule: bet $1, double after every loss, back to $1 after a win, stop at +$40.

Each win recovers all previous losses plus $1, so the only way to lose is a run of losses the bankroll cannot cover. With no limit on the bankroll every one of 2,000 simulated plays reached +$40, median 84 spins. The expected value per spin is still negative; it hides in the size of the bet needed to climb out of a run (Table 5).

Give the player a bankroll and sweep it (Table 6):

Check one row by hand: with $255 you either reach +$40 or lose all $255, so $0.791 \times 40 - 0.208 \times 255 = 31.6 - 53.0 = -21.4$, close to the simulated −$21.5. A bigger bankroll raises the hit rate from 49% to 97% and makes the median play a winner, and the mean stays negative the whole way (the third panel of Figure 1). Each doubling of the bankroll makes the rare loss twice as big, but it takes one more loss in a row to get there, which multiplies its probability by only 0.5263, a little more than a half. Twice the loss at slightly more than half the chance is a slightly bigger expected loss. The expected value of the game never changed. A high hit rate with a negative mean is a strategy that wins small and loses big; it is the shape of selling insurance, of selling options, and of the pair trade above when the pair breaks. The median hides the tail; the mean shows it; and only a measure of the tail, such as drawdown or the Sortino ratio, tells you the two apart.

Traps

Table 7 is the list of mistakes, with the fix beside each.

The code

The SPY checks, the rule and its costs, the pair’s z-score, and the doubling rule. It reads the blog’s price file from the examples repo, so it runs as pasted (pip install numpy pandas). r holds the daily returns; spy SPY’s column; rule is 1 on the days the rule holds SPY and 0 on cash days; trades is 1 on each day the position changes.

import numpy as np
import pandas as pd

URL = "https://raw.githubusercontent.com/ankurtrapasiya/examples/main/data/prices.csv"
prices = pd.read_csv(URL, index_col="date", parse_dates=True)
r = prices.pct_change().dropna()                 # daily returns, one column per ticker
spy = r["SPY"]

print(round(spy.autocorr(1), 3))                 # -0.158: correlation of today with yesterday
up = spy > 0
print(round(up[up.shift(1) == False].mean(), 3)) # 0.568: P(up | yesterday down)

rule = (~up).astype(float).shift(1).fillna(0)    # 1 = hold SPY today because yesterday was down
print(f"{(1 + rule * spy).prod() - 1:+.1%}")     # +115.4% before costs
trades = rule.diff().abs().fillna(0)             # 1 on each day the position changes
print(int(trades.sum()))                         # 747 position changes
for cost in (0.001, 0.005):
    print(f"{cost:.1%} a change: {(1 + rule * spy - trades * cost).prod() - 1:+.0%}")

ratio = np.log(prices["GS"] / prices["XOM"])    # the pair's price ratio, on a log scale
z = (ratio - ratio.rolling(60).mean()) / ratio.rolling(60).std()   # the pair's 60-day z-score
print(round(z.min(), 1), round(z.max(), 1), int((z.abs() > 2).sum()))   # -4.6 4.1 169

def play(rng, bankroll=None, p=18/38, target=40):   # the doubling rule, one play
    bet, total = 1, 0
    for _ in range(2000):                        # at most 2,000 spins
        if total >= target or (bankroll is not None and bankroll + total <= 0):
            break                                # reached +$40, or the bankroll is gone
        if bankroll is not None:
            bet = min(bet, bankroll + total)     # you cannot bet more than you have left
        if rng.random() < p: total += bet; bet = 1     # win: back to $1
        else:                total -= bet; bet *= 2    # loss: double the bet
    return total

rng = np.random.default_rng(0)
finals = np.array([play(rng, bankroll=255) for _ in range(2000)])
print(f"bankroll $255: reached +$40 {np.mean(finals >= 40):.1%}, mean final {finals.mean():+.1f}")

It prints −0.158 and 0.568, then +115.4% and the 747 position changes, then the two cost rows (+2.0% and −94.9%), then the z-score’s range and the 169 days beyond ±2, then one simulated sweep at a $255 bankroll (about 80% reaching +$40, a negative mean; a different random seed gives slightly different numbers than the table). (~up) flips True and False, so it marks the down days; .shift(1) moves each value to the next day, so today’s position uses yesterday’s direction.

Summary

Table 8 collects the rules.

References

• Eugene F. Fama, “Efficient Capital Markets: A Review of Theory and Empirical Work”, The Journal of Finance 25(2), 383–417, 1970, the three forms
• Philip J. Romero and Tucker Balch, What Hedge Funds Really Do: An Introduction to Portfolio Management (Business Expert Press, 2014), chapters 3 and 8: arbitrage in the broad sense, efficiency as reaction speed, and why so few managers beat their index
• Ernest P. Chan, Quantitative Trading: How to Build Your Own Algorithmic Trading Business (Wiley, 2009), chapter 7: mean reversion, momentum, and cointegration against correlation