Mean and standard deviation: return and risk in two numbers

Any analysis of returns starts with these two numbers. Almost every measure of a fund or strategy, including the Sharpe ratio, is built from them. The expected value post ended with “two choices can have the same average and very different risk”. This post measures that difference.

First: daily returns

In finance we rarely work with prices directly. We work with returns: the percentage change from one day to the next. This is the definition in Romero and Balch’s What Hedge Funds Really Do (chapter 2):

returns, the datathe meanthe spread: standard deviation, variance

$$\good{r_t} = \frac{p_t}{p_{t-1}} - 1$$

SPY (the S&P 500 fund) closed 2019 at $223.25 on its first trading day and $217.92 the next: $r = 217.92/223.25 - 1 = -0.0239 = \good{-2.4\%}$. Returns let you compare a $10 stock with a $1,000 stock on the same scale. The examples below use five years of daily closes for five tickers: SPY, AAPL, JPM, XOM and GLD (gold), 2019 to 2023, adjusted for dividends and splits.

Daily returns compound: to get the return over many days, multiply one plus each day’s return together and subtract one (same chapter):

$$\text{cumulative return} = \prod_{t=1}^{n} (1 + \good{r_t}) - 1$$

Read $\prod$ as “multiply together”, the way $\sum$ means “add together”. That’s why the two funds later in this post end the year at different values despite the same average day, and why $100 in AAPL became $508 over the five years while the same $100 in GLD became $158.

The mean: where the middle is

Plain meaning: add everything up, divide by how many there are.

$$\key{\bar{r}} = \frac{1}{n} \sum_{t=1}^{n} \good{r_t}$$

SPY’s first five daily returns of 2019, rounded to one decimal: −2.4%, 3.3%, 0.8%, 0.9%, 0.5%.

$$\key{\bar{r}} = \frac{-2.4 + 3.3 + 0.8 + 0.9 + 0.5}{5} = \frac{3.1}{5} = 0.62\%$$

On an average day that week, SPY made $\key{0.62\%}$. (A wild week: the five-year average daily return is 0.066%.)

The standard deviation: how far things stray

Plain meaning: how far, typically, a day’s return lands from the average.

The recipe:

1. For each day, find how far it is from the mean (the deviation).
2. Square each deviation, so negatives don’t cancel positives.
3. Average the squares. This is the variance.
4. Take the square root, to get back to the original units (percent). This is the standard deviation.

$$\risk{s} = \sqrt{\frac{1}{n-1} \sum_{t=1}^{n} \left( \good{r_t} - \key{\bar{r}} \right)^2}$$

Worked out for the five days (mean 0.62%):

$$\begin{aligned} \risk{s^2} &= \frac{16.43}{5 - 1} \\[10pt] &= 4.11 \end{aligned}$$ $$\begin{aligned} \risk{s} &= \sqrt{4.11} \\[10pt] &= 2.03\% \end{aligned}$$

A typical day that week landed about 2 percentage points away from the 0.62% average. Over the whole five years SPY’s daily standard deviation is 1.32%; that first week of January 2019 was twice as jumpy as normal.

Why divide by $n - 1$, not $n$?

This question comes up every time. When you compute the deviations, you measure them from the sample’s own mean, which is pulled towards the data you happen to have. So the deviations come out a little too small on average. Bishop and Bishop’s Deep Learning: Foundations and Concepts shows exactly how much (equation 2.60): the divide-by-$n$ variance comes out, on average, at

$$E\left[\risk{\sigma^2_{\text{ML}}}\right] = \frac{n-1}{n}\,\risk{\sigma^2}$$

of the true variance $\risk{\sigma^2}$. Multiplying by $n/(n-1)$ undoes that, which is the same as dividing by $n - 1$ instead of $n$ (their equation 2.63). It’s called Bessel’s correction, and the result is the sample standard deviation.

Dividing by $n$ gives the population standard deviation, which is right only when you have every possible value, not a sample. Romero and Balch’s What Hedge Funds Really Do (chapter 2) writes volatility with $1/n$; with years of daily data the two versions are practically equal. Here: 2.03% (sample) versus 1.81% (population). With a year of daily data the difference is tiny, but with five days it’s large.

A practical trap: NumPy’s np.std divides by $n$ by default. Pandas’ .std() divides by $n - 1$. Use np.std(x, ddof=1) to get the sample version in NumPy.

Five years, five tickers

The same two numbers for each ticker over 2019 to 2023, daily and then annualised (the next section explains the 252 and the square root):

Two things to notice. Gold is the calm one: the smallest average and the smallest spread. And JPM and XOM earned about the same average day as each other with the same risk, yet JPM ended higher; the swings along the way differed, which is the next section’s point.

Same mean, different risk

To isolate that point I made up two funds with exactly the same average daily return, 0.04%, but different standard deviations: 0.5% for the calm one, 1.5% for the wild one.

Same average, but the wild fund ended lower, and not by bad luck. Losses hurt more than equal gains help: lose 10% and then gain 10%, and you’re at 99%, not 100%. The more a fund swings, the more this “volatility drag” eats into growth. A rough rule: the yearly growth rate is about the average return minus half the variance.

From daily to yearly: why √252

There are about 252 trading days in a year. To turn daily numbers into yearly ones, with $\key{\mu}$ for the mean and $\risk{\sigma}$ for the standard deviation:

$$\key{\mu_{\text{annual}}} \approx 252 \times \key{\mu_{\text{daily}}}$$ $$\risk{\sigma_{\text{annual}}} \approx \sqrt{252} \times \risk{\sigma_{\text{daily}}}$$

The mean grows with the number of days, but the standard deviation grows with the square root. Why?

If days are independent, their variances add up. Over 252 days:

$$\risk{\text{Var}}(\text{year}) = 252 \times \risk{\text{Var}}(\text{day})$$ $$\begin{aligned} \risk{\text{SD}}(\text{year}) &= \sqrt{252 \times \risk{\text{Var}}(\text{day})} \\[10pt] &= \sqrt{252} \times \risk{\text{SD}}(\text{day}) \end{aligned}$$

Good days and bad days partly cancel out, so risk grows more slowly than time. I checked it by simulating 20,000 years of independent days with a 1% daily standard deviation. The standard deviation of the yearly totals came out at $\risk{15.86\%}$. The formula says $\sqrt{252} \times 1\% = 15.87\%$.

For our funds: the calm one is $0.5\% \times \sqrt{252} = 7.94\%$ a year, the wild one $\risk{23.81\%}$.

The same $\sqrt{252}$ shows up in the Sharpe ratio, which has its own post later in this series.

The 68–95–99.7 rule, and why markets break it

If returns follow a normal distribution (the bell curve), the standard deviation tells you a lot:

Real market returns are not quite normal. They have fat tails: extreme days happen far more often than the bell curve says. I simulated a million days from a fat-tailed distribution (Student’s t with 3 degrees of freedom) with the same standard deviation:

About 77 times more extreme days (0.62 ÷ 0.008). A risk model that assumes the bell curve will badly underestimate how often a crash day comes. The standard deviation is a good summary of everyday risk, and a poor guide to the worst days.

The code

import numpy as np
import pandas as pd

prices = pd.read_csv("prices.csv", index_col="date", parse_dates=True)   # adjusted closes
returns = prices / prices.shift(1) - 1                                   # r_t = p_t / p_{t-1} - 1

spy = returns["SPY"].dropna()
mean = spy.mean()                               # 0.00066  (0.066% a day)
sd = spy.std()                                  # 0.0132   (1.32%), pandas divides by n - 1
sd_pop = spy.std(ddof=0)                        # numpy's default, divides by n

annual_mean = mean * 252                        # 16.7%
annual_sd = sd * np.sqrt(252)                   # 21.0%
growth = 100 * (1 + spy).prod()                 # $100 -> $206 over 2019-2023

Summary

Next

Covariance and correlation: how two assets move together, and why combining two risky assets can give you something less risky than either one.

References

• Philip J. Romero and Tucker Balch, What Hedge Funds Really Do: An Introduction to Portfolio Management (Business Expert Press, 2014), chapter 2: returns, cumulative return, volatility
• Christopher M. Bishop and Hugh Bishop, Deep Learning: Foundations and Concepts (Springer, 2024), chapter 2, section 2.3: why dividing by $n$ underestimates variance (equation 2.60) and the $n - 1$ correction (2.63)
• Yves Hilpisch, Python for Finance: Mastering Data-Driven Finance (2nd ed., O’Reilly, 2018), chapter 8, “Financial Time Series”
• NumPy std, for the ddof setting