Evaluating a strategy: where the return came from

The backtesting post built the machine that turns a rule into a history of returns, and the SMA cross post looked at one rule in detail: hold while the 50-day average close is above the 200-day average, step aside when it falls below. This post takes the output of that machine and asks what it proves. The rule here runs on all five of the blog’s tickers (SPY, NVDA, GS, XOM, GLD) at once: each ticker gets a 20% slice of the money, the slice is held while that ticker’s 50-day average is above its 200-day average at the previous close and sits in cash otherwise, and every switch pays 0.1% of the money moved. The test runs from January 2021, the first month in which every average used below exists, to October 2025. $10,000 became $31,560, a return of $\good{+215.6\%}$. The rest of the post is about why that number, on its own, is not evidence.

The benchmark decides what “good” means

A return needs something to be compared with. Romero and Balch, in What Hedge Funds Really Do, define a benchmark as “a nonmanaged investment that presents a relevant comparator”, and add that a strategy narrower than the whole market should be compared with a narrower index, or a weighted mix of indexes matching what it holds. Ernest Chan’s Quantitative Trading gives the same rule in examples: a long-only stock strategy that makes 10% a year is “not too fantastic” because an index fund does as well, a strategy that trades small companies is measured against a small-company index rather than the S&P 500, and a long-short strategy with equal money on each side is measured against a riskless asset such as a Treasury bill.

The crossover rule can be set against three candidates. Table 1 has them on the same dates.

Against cash the rule is $\good{215.6}$ points ahead, and against SPY $\good{120.8}$ points ahead. Against the five tickers simply held, it is $\bad{111.8}$ points behind. Only the last comparison answers the question the rule poses. The rule does not choose what to own: the five tickers were chosen before it ran, and all it decides is when to hold each one. So the passive version of the same decision is “hold all five all the time”, and the rule’s value is whatever it adds to that. Cash and SPY are both beaten by the choice of tickers, not by the rule. (Cash is counted as earning nothing here; a real fund’s idle cash earns the Treasury bill rate, which would add a little to the rule’s return, because the rule sits in cash a quarter of the time on average.)

Pick the benchmark from the decision the strategy makes, before seeing any results. A benchmark chosen after the fact is usually the one the strategy beat.

Four numbers, read together

Romero and Balch measure risk as the standard deviation of returns, the volatility, and give the usual objection to it: it “does not differentiate upward deviations … from downward deviations”. Drawdown is the measure they offer next to it, “simply the maximum drop from peak to trough, measured as a percentage of the peak level”. In a formula, the drawdown on day $t$ is the portfolio’s value that day divided by the highest value it has reached so far, minus one:

DD, the drawdown on day tV, the portfolio's value

The maximum drawdown is the most negative value of $\risk{\text{DD}_t}$ over the whole test. Chan calls the running maximum the “high watermark”, and adds a second number that matters to anyone living through it: how long the portfolio took to get back above that mark.

Read across Table 1, the four columns tell one story. The rule’s volatility was $\risk{14.5\%}$ against $\risk{19.3\%}$ for holding the five, and its worst fall was $\risk{-13.0\%}$, from 18 February to 8 April 2025, against $\risk{-22.1\%}$ for the five held, whose worst point was 26 September 2022, and $\risk{-24.5\%}$ for SPY, from 3 January to 12 October 2022. The Sharpe ratio, return per unit of volatility as the Sharpe, Sortino and information ratio post set it up, came out almost equal: $\key{1.72}$ for the rule and $\key{1.66}$ for holding. The rule gave up about a quarter of the yearly return (26.9% against 35.2%), and took a quarter off the volatility and two fifths off the worst fall. That is a trade, and whether it is a good one depends on the investor, not on the backtest.

The fourth column, turnover, is how much of the portfolio is bought or sold, added up over a year: 162% means the rule traded, in total, 1.6 times the portfolio’s value every year. It came from 35 switches of a ticker in or out, and at 0.1% a switch it cost 0.78 percentage points of return over the whole test, small enough that the rule before costs (Sharpe 1.73) and after (1.72) look the same. That is a property of a slow rule, not of rules in general. Chan’s warning is that “a strategy with a high Sharpe ratio before adding transaction costs can become very unprofitable after adding such costs”, and the what a trade costs post has the model.

A Sharpe ratio over five years is one number, and it hides how uneven the years were (Table 2).

The rule’s Sharpe ratio ran from $\key{0.30}$ to $\key{2.55}$ depending on the year, and it beat holding in exactly one of the five, 2022, the year with the fall in it. Romero and Balch show a backtest report with a Sharpe ratio of 0.50 against 0.27 for the market, and warn soon after that “it is easy to overinterpret positive results”. Four years out of five, holding was better here.

Where the return came from

The next question is what produced the $\good{+215.6\%}$. Romero and Balch describe the capital asset pricing model as “the most basic disaggregation of an investment’s performance into two parts”: the part “based on market return (which could be achieved simply by buying an index)”, which is beta, and the part based on “company-specific factors”, which is alpha. The beta and alpha post fitted that split to single stocks; here it is fitted to the rule’s daily returns, with SPY as the market:

r, the rule's return on day tα, what the market does not explain

The rule’s beta was 0.58, and SPY explained 47% of its daily variation. For attribution the returns are averages of daily returns times 252, because averages add up across parts and compound returns do not, so the numbers in this section are a little lower than the compound +26.9% a year of Table 1. The rule averaged 24.9% a year; the market part, beta times SPY’s average of 15.3%, was 8.8%; the intercept, the alpha, was $\key{+16.1\%}$ a year.

Is that alpha real? An estimated coefficient comes with a standard error, and ISLP (section 3.1.2) tests it with the t-statistic, the estimate divided by its standard error, the number of standard errors it sits away from zero:

α, the estimated alpha

The alpha’s standard error was 4.8% a year, so $t = \key{3.33}$. ISLP gives cut-offs of “around 2 and 2.75” for the usual 5% and 1% levels (with 30 observations), the chance of an estimate that far from zero when the true value is zero. By that test the rule has skill.

It does not. Split the same average return by ticker instead of by market, and compare each slice with the same slice simply held (Table 3, and the third panel of Figure 1):

The timing column is negative on every ticker. Stepping aside when the averages crossed cost $\bad{6.9}$ points a year of average return, plus 0.16 points of trading costs, which takes the 25.1% to the 24.9% above. The $\key{+16.1\%}$ “alpha” against SPY did not come from the rule: the rule’s own timing subtracted on every ticker, so what SPY could not explain was which five tickers were held, and half the rule’s return came from one of them, NVDA. Take NVDA out and run the same rule on the other four, 25% each: its Sharpe ratio falls to $\key{1.27}$, against $\key{1.46}$ for holding those four. A high Sharpe ratio can be one exposure that happened to pay, and here it was.

This is the three-part split that attribution aims for: the part that was simply the market, the part from known exposures, here the four sectors the tickers stand for (technology, banks, energy, gold), and a residual that might be skill. The residual is what is left once the second part is removed, and on this rule it is negative. With a real universe of hundreds of stocks the middle step is the same regression with more columns, one for each known driver of returns; the CFA Institute’s Handbook of Artificial Intelligence and Big Data Applications in Investments (chapter 1) names Fama and French (1993) as an example of such published drivers. Each column added takes return away from alpha and gives it to an exposure anyone could buy.

Alpha is measured against a model, and it shrinks as the model learns what the strategy holds. Against SPY this rule had a $t$ of 3.33; against the five tickers it trades, its timing lost money.

Testing on data the rule never saw

Every number so far was measured on the period the rule was looked at in. That is the first of the three cautions Romero and Balch list in their chapter on backtesting: the designer knows how history turned out, and can choose a strategy “at least unconsciously, designed for those conditions”. Their fix is to keep the period that inspired the strategy apart from the period that tests it: “Otherwise, you are simply confirming a tautology.” Chan’s version is a split. Tune every parameter and every design choice on the first part of the data, test the result once on the second, more recent part, and never adjust anything to improve the second part, because “you have effectively used the test set as your training set”. It is the same discipline as the train/test split in the scoring a model post.

So: choose the two windows on 2021 to 2023, then test on 2024 and 2025. Eighty pairs, fast windows of 10 to 100 days and slow windows of 50 to 250 days, were scored on their Sharpe ratio net of costs in the first period (Table 4).

The best pair in-sample came 38th of 80 on the unseen years. Across all eighty, the rank correlation between the two periods’ Sharpe ratios was $\key{-0.13}$, which is to say the ranking on 2021 to 2023 said nothing about the ranking on 2024 to 2025. The search found the pair that fit those three years best; it did not find a better rule. The out-of-sample Sharpe ratios are also higher than the in-sample ones for almost every pair (the median was 2.06), and that is not the rule improving: holding the five went from 1.33 to 2.24 between the two periods. The unseen period measured the market of 2024 and 2025 first and the rule second, which is why the comparison with the benchmark on the same dates matters even out of sample.

A test period is only clean until you act on it. Look at the 2024 to 2025 numbers, change the windows, and those years have become part of the search.

Three traps, and the standard defence

Romero and Balch name three pitfalls, and the rule above walked into all three.

• Peeking into the future. The five tickers were chosen after the period was over, knowing that NVDA was one of its great winners. A rule run on a universe picked with hindsight inherits the hindsight, which is what Table 3 found. The CFA handbook makes the same point from the other side: test over the dot-com crash without technology stocks in the universe, and “it is incumbent upon the investor to provide a sensible reason why”.
• Data mining. Test enough variants and one will look good by chance; Romero and Balch’s test is whether there is “any plausible theory to explain why a given strategy worked”. The 80 window pairs of Table 4 are a small version, and the data that lies post runs a thousand coin-flip signals to show the large one. The CFA handbook’s habit against it is to write down every variant tried, “including those that failed to show good results”.
• A changing market. The 2021 to 2023 market held a bear year; 2024 and 2025 rose strongly, with one sharp fall in April 2025. Chan calls these “regime shifts” and points out that more data helps only “when the financial time series is generated by a stationary process”, which markets are not. The CFA handbook adds that published research itself changes how investors behave, and so changes the market it described.

The defence Romero and Balch give is cross-validation: “testing the strategy on more than one subset of historical data to see if results are similar in each subset”, either two sets of stocks or two periods. Table 5 runs the 50/200 rule on both.

The result is consistent, just not in the direction the headline suggested. On all five and on the three stocks the rule had the better Sharpe ratio in 2021 to 2022, with the fall in it, and the worse one in 2023 to 2025, a rising market; on the two funds it was a little worse in both. Its drawdown was smaller in five of the six cells and equal in the sixth. Romero and Balch’s reading is that strong results in only one subset are “likely” not valid. What survives every subset here is the smaller fall; the extra return depends on the period containing a decline long enough for a 200-day average to see it.

Traps

Table 6 sets the common beliefs about backtest results against what the five tickers showed.

The code

The snippet reproduces the rule’s row and the five-held row of Table 1, and the beta and alpha against SPY. Table 7 names its variables.

import numpy as np
import pandas as pd

url = "https://raw.githubusercontent.com/ankurtrapasiya/examples/main/data/prices.csv"
px = pd.read_csv(url, index_col="date", parse_dates=True)[["SPY", "NVDA", "GS", "XOM", "GLD"]]
r = (px / px.shift(1) - 1).dropna()

on = (px.rolling(50).mean() > px.rolling(200).mean()).astype(float)
w = (on / 5).shift(1).loc["2021":]        # 20% slice per ticker, decided at yesterday's close
r = r.loc["2021":]
turnover = w.diff().fillna(w).abs().sum(axis=1)   # share of the money bought or sold each day
rule = (w * r).sum(axis=1) - 0.001 * turnover
held = r.mean(axis=1)                     # the same five, 20% each, always held

def report(name, x):
    value = (1 + x).cumprod()
    sharpe = np.sqrt(252) * x.mean() / x.std()
    worst = (value / value.cummax() - 1).min()   # drawdown: fall from the highest value so far
    print(f"{name:<12} total {value.iloc[-1] - 1:+.1%}  Sharpe {sharpe:.2f}  max drawdown {worst:.1%}")

report("50/200 rule", rule)
report("five held", held)
beta, alpha = np.polyfit(r["SPY"], rule, 1)   # slope and intercept of the rule against SPY
print(f"beta {beta:.2f}, alpha {252 * alpha:+.1%} a year")

It prints a total of +215.6%, a Sharpe ratio of 1.72 and a maximum drawdown of −13.0% for the rule; +327.4%, 1.66 and −22.1% for the five held; and a beta of 0.58 with an alpha of +16.1% a year. .shift(1) is what keeps the rule honest: each day’s weight comes from the averages at the previous close. .fillna(w) counts the first day’s purchases as turnover, and np.polyfit(x, y, 1) fits a straight line and returns its slope and intercept.

Summary

Table 8 collects the ideas, with the five-ticker number behind each.

References

• Philip J. Romero and Tucker Balch, What Hedge Funds Really Do: An Introduction to Portfolio Management (Business Expert Press, 2014). Chapter 14 for the backtest report against a benchmark, the warning against overinterpreting positive results, the three pitfalls (peeking into the future, data mining, a changing market) and cross-validation on subsets of stocks or periods; chapter 2 for the benchmark as a relevant comparator, volatility as the standard deviation of returns and drawdown as the maximum drop from peak to trough; chapter 7 for the capital asset pricing model as the split of performance into a market part and a company-specific part; chapter 9 for alpha as the residual and the information ratio computed on it.
• Ernest P. Chan, Quantitative Trading: How to Build Your Own Algorithmic Trading Business (Wiley, 2009). Chapter 2, “How Does It Compare with a Benchmark and How Consistent Are Its Returns?” and “How Deep and Long Is the Drawdown?”, for matching the benchmark to what the strategy trades, the information ratio and the high watermark, and the passage on regime shifts and non-stationary prices; chapter 3 for out-of-sample testing, not tuning on the test set, and transaction costs turning a high Sharpe ratio unprofitable.
• CFA Institute Research Foundation, Handbook of Artificial Intelligence and Big Data Applications in Investments (2023). Chapter 1, by Mike Chen and Weili Zhou, for overfitting giving “a fantastic backtest”, the universe chosen with hindsight as a form of lookahead bias, documenting every tested variant, and markets that change as investors learn, with Fama and French (1993) as an example of published drivers of returns.
• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 3.1.2: the t-statistic as the estimate over its standard error, with cut-offs of around 2 and 2.75.