Portfolio optimisation: choosing the weights

Portfolio optimisation is how you decide how much money to put in each asset. This post explores the idea by drawing thousands of random portfolios, which shows the shape of the problem and makes the intuition easier to see than an optimiser would.

This post pulls together the ideas before it in the Quantitative Finance Series 1: it needs expected returns, standard deviations, correlations and the Sharpe ratio.

The question

You have money to split across five tickers. Each has an expected yearly return, a standard deviation, and correlations with the others. What mix should you hold?

The inputs are the 2019 to 2023 figures from the earlier posts, annualised, with a 2% risk-free rate:

The stocks move together (correlations 0.3 to 0.8, from the correlation post). Gold is nearly unrelated to the rest (0.1 or less). Keep one thing in mind throughout: “expected return” here means “what happened in those five years”, which is the standard shortcut and, as the end of the post shows, the weak point.

The two formulas you need

For weights $\key{w}$ (one per asset, adding up to 1), expected returns $\good{\mu}$ and covariance matrix $\risk{\Sigma}$. Romero and Balch’s What Hedge Funds Really Do (chapter 10) writes these as $\good{r_p} = \sum_i \key{w_i} \good{r_i}$ and $\risk{\sigma_p^2} = \sum_i \sum_j \key{w_i} \key{w_j} \risk{\sigma_i} \risk{\sigma_j} \rho_{ij}$; in vector form:

the weights we choosereturnsrisk

$$\text{expected return: } \good{\mu_p} = \key{w^\top} \good{\mu}$$ $$\text{risk: } \risk{\sigma_p} = \sqrt{\key{w^\top} \risk{\Sigma}\, \key{w}}$$

The return is just the weighted average of the assets’ returns. The risk is not a weighted average: as the correlation post showed, the correlations inside $\risk{\Sigma}$ decide how much of the risk cancels out. That asymmetry is the reason optimisation is worth doing at all.

50,000 random portfolios

I drew 50,000 random long-only mixes (all weights between 0 and 1, adding up to 1) and computed the return and risk of each:

Three things to notice:

1. The cloud has a sharp upper-left edge. No mix gets above it. That edge is the efficient frontier: for each level of risk, the highest expected return available. Anything below the edge is beaten by a portfolio straight above it, with the same risk and more return.
2. The lowest-risk mix (blue) has 12.7% risk, less than gold alone (14.9%). Adding some SPY and a little JPM to gold lowers risk, because they don’t move together.
3. The best mix beats every single asset on Sharpe ratio. The best of the 50,000 has a Sharpe of $\key{1.20}$, higher than AAPL alone (1.11), the best single asset.

Notice what the best mix is: the stock with the best past return plus the asset that moves independently of it. Nothing in SPY, even though SPY is “the market”: it’s just a worse version of AAPL in these years, and AAPL plus gold diversifies better.

(Random search only finds points near the true optimum. An optimiser such as SciPy’s minimize on the negative Sharpe ratio would find the exact edge. With 50,000 draws in five assets, we’re close.)

What “optimal” means

Romero and Balch state the problem like this: choose the weights with the least risk that still meets a target return $r^*$, fully invested:

$$\begin{array}{ll} \underset{\key{w}}{\text{minimise}} \quad & \risk{\sigma_p}(\key{w}) \\[12pt] \text{subject to} \quad & \good{r_p}(\key{w}) \ge r^*,\; \textstyle\sum_i \key{w_i} = 1 \end{array}$$

Solve it once for every possible target $r^*$, and the answers trace out the efficient frontier: the lowest-risk portfolio for each achievable return. In practice you add limits too, such as no more than 40% in any one asset (with five assets, a cap must be at least 20% each, or the weights cannot add up to 1). Two special cases come up again and again:

Minimum variance: the mix with the least risk, whatever the return.

$$\begin{array}{ll} \underset{\key{w}}{\text{minimise}} \quad & \key{w^\top} \risk{\Sigma}\, \key{w} \\[12pt] \text{subject to} \quad & \textstyle\sum_i \key{w_i} = 1 \end{array}$$

If short selling is allowed (weights can be negative), this has an exact answer:

$$\key{w_{\text{min var}}} = \frac{\risk{\Sigma^{-1}} \mathbf{1}}{\mathbf{1}^\top \risk{\Sigma^{-1}} \mathbf{1}}$$

where $\mathbf{1}$ is a vector of ones. Note that it doesn’t use expected returns at all, which turns out to be an advantage (see below).

Maximum Sharpe ratio: the mix with the most excess return per unit of risk.

$$\begin{array}{ll} \underset{\key{w}}{\text{maximise}} \quad & \displaystyle\frac{\key{w^\top} \good{\mu} - r_f}{\sqrt{\key{w^\top} \risk{\Sigma}\, \key{w}}} \\[12pt] \text{subject to} \quad & \textstyle\sum_i \key{w_i} = 1,\; \key{w_i} \ge 0 \end{array}$$

Without the long-only limit, the answer points in the direction $\risk{\Sigma^{-1}}(\good{\mu} - r_f \mathbf{1})$, rescaled so the weights add up to 1. With limits like $\key{w_i} \ge 0$ there’s no neat formula, and you need a numerical optimiser.

The capital market line

The dashed line on the chart starts at the risk-free rate (2%, zero risk) and just touches the cloud at the best-Sharpe portfolio. Its slope is that Sharpe ratio:

$$\good{E[R]} = r_f + \text{Sharpe}_{\text{best}} \times \risk{\sigma}$$

Every point on the line is a mix of the best-Sharpe portfolio and cash. Want less risk? Hold some cash. More? Borrow (if you can). The line lies above the frontier everywhere except where it touches, so in theory everyone should hold the same risky mix, and only change how much cash they hold beside it. It is a surprising result: the choice of risk becomes a choice of how much cash to hold, not of which assets.

An optimiser instead of random draws

Random search finds a point near the best mix. A numerical optimiser finds the mix itself, and naming its four parts is most of understanding it:

Three names that go with the frontier: the global minimum variance portfolio is the lowest-risk point on it and needs no return forecasts at all, only the covariance matrix, which makes it the most robust; the tangency portfolio is the highest-Sharpe point, where the capital market line touches; and moving along that line, mixing the tangency portfolio with cash, is how an investor chooses risk without changing the mix.

Chosen on one period, run on the next. The optimiser on the five tickers, fitted to one year and then held through the following year, against the 20%-each portfolio:

In 2022, the one year where a single asset had a strongly positive Sharpe (XOM at 1.88 while SPY and AAPL were negative), the optimiser put everything in it, because any diversification would have lowered the in-sample ratio; the next year that portfolio made −2.9% while equal weights made +24.0%. In every window the in-sample Sharpe at least halved out of sample. Equal weights won four of the five.

Why real optimisers disappoint

The maths is exact. The inputs are not. Expected returns are very hard to estimate: a few years of data tell you almost nothing reliable about a stock’s average return. AAPL’s 37.8% a year is what happened in 2019 to 2023, not what anyone could have forecast in 2018, and nobody should expect it to repeat. And the optimiser takes your guesses completely literally.

I changed one input: AAPL expected to return 32.8% instead of 37.8%, still a great stock. Everything else the same.

A five-point change in a number nobody can forecast to five points moved an eighth of the portfolio. Nudge JPM’s return up by two points instead and JPM goes from 0% to 7%. Optimisers concentrate money in whatever assets have the most optimistic estimates, which are often the ones with the biggest estimation errors. Richard Michaud called mean-variance optimisers “error maximizers”.

What practitioners do about it:

• Prefer minimum variance, which doesn’t need expected returns at all.
• Add limits: no asset above some maximum, or close to a benchmark.
• Shrink estimates towards something sensible, like equal returns, before optimising.
• Check stability: if a small change to the inputs flips the answer, don’t trust the answer.

The code

This only scores random portfolios; it doesn’t search for the best one. It reads the blog’s price file from the examples repo, so it runs as pasted (pip install numpy pandas). mu holds the five expected yearly returns and cov the yearly covariance matrix; W holds 50,000 random mixes, one row per mix and one column per ticker.

import numpy as np
import pandas as pd

URL = "https://raw.githubusercontent.com/ankurtrapasiya/examples/main/data/prices.csv"
prices = pd.read_csv(URL, index_col="date", parse_dates=True)   # SPY, AAPL, JPM, XOM, GLD
r = (prices / prices.shift(1) - 1).dropna()     # daily returns
mu = (r.mean() * 252).to_numpy()                # expected yearly returns (the five-year averages)
cov = (r.cov() * 252).to_numpy()                # yearly covariance matrix

W = np.random.default_rng(17).dirichlet(np.ones(5), 50_000)   # 50,000 random long-only weights
ret = W @ mu                                                  # each portfolio's return
risk = np.sqrt(np.einsum("ij,jk,ik->i", W, cov, W))           # sqrt(w' cov w) for each row
sharpe = (ret - 0.02) / risk                                  # 2% risk-free rate
best, safest = sharpe.argmax(), risk.argmin()
mix = lambda w: {t: round(float(x), 2) for t, x in zip(prices.columns, w)}  # ticker -> weight
print(mix(W[best]), f"Sharpe {sharpe[best]:.3f}, return {ret[best]:.1%}, risk {risk[best]:.1%}")
print(mix(W[safest]), f"risk {risk[safest]:.1%}")

It prints the best mix (53% AAPL, 42% GLD, 5% XOM) with its Sharpe of about 1.20, return 25.3% and risk 19.3%, then the lowest-risk mix (70% GLD, 22% SPY, 7% JPM) at 12.7%. dirichlet draws random weights that are never negative and always add up to 1. einsum("ij,jk,ik->i", ...) computes $w^\top \Sigma w$ for every row of W at once.

The script behind all the numbers

The frontier chart and tables come from examples/portfolio-optimisation/random_portfolios.py. To run it, save it next to the shared helpers market_data.py and chartstyle.py and the price file data/prices.csv, keeping the folders as in the links, then run uv run examples/portfolio-optimisation/random_portfolios.py.

Summary

The series map puts the series’ ten ideas on one page.

References

• Philip J. Romero and Tucker Balch, What Hedge Funds Really Do: An Introduction to Portfolio Management (Business Expert Press, 2014), chapter 10: modern portfolio theory, the efficient frontier and the optimisation problem
• Harry Markowitz, “Portfolio Selection”, The Journal of Finance 7(1), 77–91, 1952
• Richard O. Michaud, “The Markowitz Optimization Enigma: Is ‘Optimized’ Optimal?”, Financial Analysts Journal 45(1), 31–42, 1989