Portfolio optimisation: choosing the weights

Portfolio optimisation is how you decide how much money to put in each asset. This post explores the idea by drawing thousands of random portfolios, which shows the shape of the problem and makes the intuition easier to see than an optimiser would.

This post pulls the whole Finance stats series together: it needs expected returns, standard deviations, correlations and the Sharpe ratio.

The question

You have money to split across five tickers. Each has an expected yearly return, a standard deviation, and correlations with the others. What mix should you hold?

The inputs are the 2019 to 2023 figures from the earlier posts, annualised, with a 2% risk-free rate:

The stocks move together (correlations 0.3 to 0.8, from the correlation post). Gold is nearly unrelated to the rest (0.1 or less). Keep one thing in mind throughout: “expected return” here means “what happened in those five years”, which is the standard shortcut and, as the end of the post shows, the weak point.

The two formulas you need

For weights $\key{w}$ (one per asset, adding up to 1), expected returns $\good{\mu}$ and covariance matrix $\risk{\Sigma}$. Romero and Balch’s What Hedge Funds Really Do (chapter 10) writes these as $\good{r_p} = \sum_i \key{w_i} \good{r_i}$ and $\risk{\sigma_p^2} = \sum_i \sum_j \key{w_i} \key{w_j} \risk{\sigma_i} \risk{\sigma_j} \rho_{ij}$; in vector form:

the weights we choosereturnsrisk

$$\text{expected return: } \good{\mu_p} = \key{w^\top} \good{\mu}$$ $$\text{risk: } \risk{\sigma_p} = \sqrt{\key{w^\top} \risk{\Sigma}\, \key{w}}$$

The return is just the weighted average of the assets’ returns. The risk is not a weighted average: as the correlation post showed, the correlations inside $\risk{\Sigma}$ decide how much of the risk cancels out. That asymmetry is the reason optimisation is worth doing at all.

50,000 random portfolios

I drew 50,000 random long-only mixes (all weights between 0 and 1, adding up to 1) and computed the return and risk of each:

Three things to notice:

1. The cloud has a sharp upper-left edge. No mix gets above it. That edge is the efficient frontier: for each level of risk, the highest expected return available. Anything below the edge is beaten by a portfolio straight above it, with the same risk and more return.
2. The lowest-risk mix (blue) has 12.7% risk, less than gold alone (14.9%). Adding some SPY and a little JPM to gold lowers risk, because they don’t move together.
3. The best mix beats every single asset on Sharpe ratio. The best of the 50,000 has a Sharpe of $\key{1.21}$, higher than AAPL alone (1.11), the best single asset.

Notice what the best mix is: the stock with the best past return plus the asset that moves independently of it. Nothing in SPY, even though SPY is “the market”: it’s just a worse version of AAPL in these years, and AAPL plus gold diversifies better.

(Random search only finds points near the true optimum. An optimiser such as SciPy’s minimize on the negative Sharpe ratio would find the exact edge. With 50,000 draws in five assets, we’re close.)

What “optimal” means

Romero and Balch state the problem like this: choose the weights with the least risk that still meets a target return $r^*$, fully invested:

$$\begin{array}{ll} \underset{\key{w}}{\text{minimise}} \quad & \risk{\sigma_p}(\key{w}) \\[12pt] \text{subject to} \quad & \good{r_p}(\key{w}) \ge r^*,\; \textstyle\sum_i \key{w_i} = 1 \end{array}$$

Solve it once for every possible target $r^*$, and the answers trace out the efficient frontier: the lowest-risk portfolio for each achievable return. In practice you add limits too, such as no more than 5% in any one asset. Two special cases come up again and again:

Minimum variance: the mix with the least risk, whatever the return.

$$\begin{array}{ll} \underset{\key{w}}{\text{minimise}} \quad & \key{w^\top} \risk{\Sigma}\, \key{w} \\[12pt] \text{subject to} \quad & \textstyle\sum_i \key{w_i} = 1 \end{array}$$

If short selling is allowed (weights can be negative), this has an exact answer:

$$\key{w_{\text{min var}}} = \frac{\risk{\Sigma^{-1}} \mathbf{1}}{\mathbf{1}^\top \risk{\Sigma^{-1}} \mathbf{1}}$$

where $\mathbf{1}$ is a vector of ones. Note that it doesn’t use expected returns at all, which turns out to be an advantage (see below).

Maximum Sharpe ratio: the mix with the most excess return per unit of risk.

$$\begin{array}{ll} \underset{\key{w}}{\text{maximise}} \quad & \displaystyle\frac{\key{w^\top} \good{\mu} - r_f}{\sqrt{\key{w^\top} \risk{\Sigma}\, \key{w}}} \\[12pt] \text{subject to} \quad & \textstyle\sum_i \key{w_i} = 1,\; \key{w_i} \ge 0 \end{array}$$

Without the long-only limit, the answer points in the direction $\risk{\Sigma^{-1}}(\good{\mu} - r_f \mathbf{1})$, rescaled so the weights add up to 1. With limits like $\key{w_i} \ge 0$ there’s no neat formula, and you need a numerical optimiser.

The capital market line

The dashed line on the chart starts at the risk-free rate (2%, zero risk) and just touches the cloud at the best-Sharpe portfolio. Its slope is that Sharpe ratio:

$$\good{E[R]} = r_f + \text{Sharpe}_{\text{best}} \times \risk{\sigma}$$

Every point on the line is a mix of the best-Sharpe portfolio and cash. Want less risk? Hold some cash. More? Borrow (if you can). The line lies above the frontier everywhere except where it touches, so in theory everyone should hold the same risky mix, and only change how much cash they hold beside it. It is a surprising result, and it is the foundation of index investing.

Why real optimisers disappoint

The maths is exact. The inputs are not. Expected returns are very hard to estimate: a few years of data tell you almost nothing reliable about a stock’s average return. AAPL’s 37.8% a year is what happened in 2019 to 2023, not what anyone could have forecast in 2018, and nobody should expect it to repeat. And the optimiser takes your guesses completely literally.

I changed one input: AAPL expected to return 32.8% instead of 37.8%, still a great stock. Everything else the same.

A five-point change in a number nobody can forecast to five points moved an eighth of the portfolio. Nudge JPM’s return up by two points instead and JPM goes from 0% to 7%. Optimisers concentrate money in whatever assets have the most optimistic estimates, which are often the ones with the biggest estimation errors. Richard Michaud called mean-variance optimisers “error maximizers”.

What practitioners do about it:

• Prefer minimum variance, which doesn’t need expected returns at all.
• Add limits: no asset above some maximum, or close to a benchmark.
• Shrink estimates towards something sensible, like equal returns, before optimising.
• Check stability: if a small change to the inputs flips the answer, don’t trust the answer.

The code

This only scores random portfolios. It doesn’t search for the best one.

import numpy as np
import pandas as pd

prices = pd.read_csv("prices.csv", index_col="date", parse_dates=True)   # SPY, AAPL, JPM, XOM, GLD
r = (prices / prices.shift(1) - 1).dropna()
mu = (r.mean() * 252).to_numpy()                # expected yearly returns (the five-year averages)
cov = (r.cov() * 252).to_numpy()                # yearly covariance matrix

W = np.random.default_rng(17).dirichlet(np.ones(5), 50_000)   # random long-only weights
ret = W @ mu
risk = np.sqrt(np.einsum("ij,jk,ik->i", W, cov, W))           # sqrt(w' cov w) for each row
sharpe = (ret - 0.02) / risk
W[sharpe.argmax()].round(2)                     # best mix among the random ones

Summary

That completes the Finance stats series: probability, expected value, mean and standard deviation, covariance and correlation, the risk-adjusted ratios, and portfolio optimisation.

References

• Philip J. Romero and Tucker Balch, What Hedge Funds Really Do: An Introduction to Portfolio Management (Business Expert Press, 2014), chapter 10: modern portfolio theory, the efficient frontier and the optimisation problem
• Harry Markowitz, “Portfolio Selection”, The Journal of Finance 7(1), 77–91, 1952
• Richard O. Michaud, “The Markowitz Optimization Enigma: Is ‘Optimized’ Optimal?”, Financial Analysts Journal 45(1), 31–42, 1989