Portfolio optimisation: choosing the weights

Portfolio optimisation is how you decide how much money to put in each asset. Drawing thousands of random portfolios shows the shape of the problem and makes the intuition easier to see than an optimiser would.

This post pulls together the ideas before it in the Quantitative Finance Series 1: it needs expected returns, standard deviations, correlations and the Sharpe ratio.

The question

You have money to split across five tickers. Each has an expected yearly return, a standard deviation, and correlations with the others. What mix should you hold?

The inputs are the 2020 to 2025 figures from the earlier posts, annualised, with a 2% risk-free rate (Table 1):

The stocks move together (correlations 0.2 to 0.8, from the correlation post). Gold is nearly unrelated to the rest (0.13 or less). Keep one thing in mind throughout: “expected return” here means “what happened in those years”, which is the standard shortcut and, as the end of the post shows, the weak point.

The two formulas you need

For weights $\key{w}$ (one per asset, adding up to 1), expected returns $\good{\mu}$ and covariance matrix $\risk{\Sigma}$. Hilpisch’s Python for Finance (chapter 13, equations 13-1 and 13-2) writes these as $\good{\mu_p} = \sum_i \key{w_i} \good{\mu_i}$ and $\risk{\sigma_p^2} = \sum_i \sum_j \key{w_i} \key{w_j} \risk{\sigma_{ij}}$, where $\risk{\sigma_{ij}}$ is the covariance of assets $i$ and $j$; in vector form:

the weights we choosereturnsrisk

$$\text{expected return: } \good{\mu_p} = \key{w^\top} \good{\mu}$$ $$\text{risk: } \risk{\sigma_p} = \sqrt{\key{w^\top} \risk{\Sigma}\, \key{w}}$$

The return is just the weighted average of the assets’ returns. The risk is not a weighted average: as the correlation post showed, the correlations inside $\risk{\Sigma}$ decide how much of the risk cancels out. That asymmetry is the reason optimisation is worth doing at all.

50,000 random portfolios

I drew 50,000 random long-only mixes (all weights between 0 and 1, adding up to 1) and computed the return and risk of each (Figure 1):

Three things to notice:

1. The cloud has a sharp upper-left edge. No mix gets above it. That edge is the efficient frontier: for each level of risk, the highest expected return available. Anything below the edge is beaten by a portfolio straight above it, with the same risk and more return.
2. The lowest-risk mix (blue) has 13.6% risk, less than gold alone (16.3%). Adding some SPY to gold lowers risk, because they don’t move together.
3. The best mix beats every single asset on Sharpe ratio. The best of the 50,000 has a Sharpe of $\key{1.61}$ (Table 2), higher than NVDA alone (1.36), the best single asset.

Notice what the best mix is: the stock with the best past return plus the asset that moves independently of it, with a little GS and XOM on the side. Nothing in SPY, even though SPY is “the market”: it’s just a worse version of NVDA in these years, and NVDA plus gold diversifies better.

(Random search only finds points near the true optimum. An optimiser such as SciPy’s minimize on the negative Sharpe ratio would find the exact edge. With 50,000 draws in five assets, we’re close.)

What “optimal” means

Romero and Balch state the problem like this: choose the weights with the least risk that still meets a target return $r^*$, fully invested:

$$\begin{array}{ll} \underset{\key{w}}{\text{minimise}} \quad & \risk{\sigma_p}(\key{w}) \\[12pt] \text{subject to} \quad & \good{r_p}(\key{w}) \ge r^* \\[4pt] & \textstyle\sum_i \key{w_i} = 1 \end{array}$$

Solve it once for every possible target $r^*$, and the answers trace out the efficient frontier: the lowest-risk portfolio for each achievable return. In practice you add limits too, such as no more than 40% in any one asset (with five assets, a cap must be at least 20% each, or the weights cannot add up to 1). Two special cases come up again and again:

Minimum variance: the mix with the least risk, whatever the return.

$$\begin{array}{ll} \underset{\key{w}}{\text{minimise}} \quad & \key{w^\top} \risk{\Sigma}\, \key{w} \\[12pt] \text{subject to} \quad & \textstyle\sum_i \key{w_i} = 1 \end{array}$$

If short selling is allowed (weights can be negative), this has an exact answer:

$$\key{w_{\text{min var}}} = \frac{\risk{\Sigma^{-1}} \mathbf{1}}{\mathbf{1}^\top \risk{\Sigma^{-1}} \mathbf{1}}$$

where $\mathbf{1}$ is a vector of ones. Note that it doesn’t use expected returns at all, which turns out to be an advantage (see below).

Maximum Sharpe ratio: the mix with the most excess return per unit of risk.

$$\begin{array}{ll} \underset{\key{w}}{\text{maximise}} \quad & \displaystyle\frac{\key{w^\top} \good{\mu} - r_f}{\sqrt{\key{w^\top} \risk{\Sigma}\, \key{w}}} \\[12pt] \text{subject to} \quad & \textstyle\sum_i \key{w_i} = 1 \\[4pt] & \key{w_i} \ge 0 \end{array}$$

Without the long-only limit, the answer points in the direction $\risk{\Sigma^{-1}}(\good{\mu} - r_f \mathbf{1})$, rescaled so the weights add up to 1. With limits like $\key{w_i} \ge 0$ there’s no neat formula, and you need a numerical optimiser.

The capital market line

The dashed line in Figure 1 starts at the risk-free rate (2%, zero risk) and just touches the cloud at the best-Sharpe portfolio. Its slope is that Sharpe ratio:

$$\good{E[R]} = r_f + \text{Sharpe}_{\text{best}} \times \risk{\sigma}$$

Every point on the line is a mix of the best-Sharpe portfolio and cash. Want less risk? Hold some cash. More? Borrow (if you can). The line lies above the frontier everywhere except where it touches, so in theory everyone should hold the same risky mix, and only change how much cash they hold beside it. It is a surprising result: the choice of risk becomes a choice of how much cash to hold, not of which assets.

An optimiser instead of random draws

Random search finds a point near the best mix. A numerical optimiser finds the mix itself, and naming its four parts is most of understanding it (Table 3):

Three names that go with the frontier: the global minimum variance portfolio is the lowest-risk point on it and needs no return forecasts at all, only the covariance matrix, which makes it the hardest to get wrong; the tangency portfolio is the highest-Sharpe point, where the capital market line touches; and moving along that line, mixing the tangency portfolio with cash, is how an investor chooses risk without changing the mix.

Chosen on one period, run on the next. The optimiser on the five tickers, fitted to one period and then held through the next, against the 20%-each portfolio (Table 4):

In 2022, when XOM had a Sharpe of 1.88 while SPY and NVDA were negative, the optimiser put everything in it, because any diversification would have lowered the in-sample ratio; the next year that portfolio made −2.9% while equal weights made +59.3%. In the first three windows equal weights won, and in two of them the in-sample Sharpe all but vanished out of sample (2.48 to 0.06, 1.88 to 0.00). Then the optimiser chose 100% NVDA, on 2023 and on 2020 to 2023, and NVDA kept running: +178.9% in 2024 against +58.2% for equal weights, +320.6% over 2024 to 2025 against +118.2% (Figure 2). It made more money and still lost on Sharpe, 2.22 against 2.37 and 1.77 against 1.99, because one stock’s risk is the whole portfolio’s risk. Equal weights won five of the six on Sharpe; the one it lost was 2024, the one window where the optimiser spread the money. All-in on one stock is not a portfolio; it is a bet that the last period repeats, and these years happen to be ones where it did.

Why real optimisers disappoint

The maths is exact. The inputs are not. Expected returns are very hard to estimate: a few years of data tell you almost nothing reliable about a stock’s average return. NVDA’s 74.9% a year is what happened in 2020 to 2025, not what anyone could have forecast in 2019, and nobody should expect it to repeat. And the optimiser takes your guesses completely literally.

I changed one input at a time: first GLD expected to return 12.5% instead of 17.5%, still a sound hedge; then, separately, NVDA 69.9% instead of 74.9%, still far ahead of everything else. Everything else the same (Table 5).

Five points off GLD moved 14 points of the money out of gold, most of it into NVDA and the rest into GS. Five points off NVDA, whose estimate is still four times anyone else’s, moved a quarter of its slot into gold and GS. No slot is safe. Optimisers concentrate money in whatever assets have the most optimistic estimates, which are often the ones with the biggest estimation errors. Richard Michaud called mean-variance optimisers “error maximizers”.

What practitioners do about it:

• Prefer minimum variance, which doesn’t need expected returns at all.
• Add limits: no asset above some maximum, or close to a benchmark.
• Shrink estimates towards something sensible, like equal returns, before optimising.
• Check stability: if a small change to the inputs flips the answer, don’t trust the answer.

The code

This only scores random portfolios; it doesn’t search for the best one. It reads the blog’s price file from the examples repo, so it runs as pasted (pip install numpy pandas). mu holds the five expected yearly returns and cov the yearly covariance matrix; W holds 50,000 random mixes, one row per mix and one column per ticker.

import numpy as np
import pandas as pd

URL = "https://raw.githubusercontent.com/ankurtrapasiya/examples/main/data/prices.csv"
prices = pd.read_csv(URL, index_col="date", parse_dates=True)   # SPY, NVDA, GS, XOM, GLD
r = (prices / prices.shift(1) - 1).dropna()     # daily returns
mu = (r.mean() * 252).to_numpy()                # expected yearly returns (the 2020 to 2025 averages)
cov = (r.cov() * 252).to_numpy()                # yearly covariance matrix

W = np.random.default_rng(17).dirichlet(np.ones(5), 50_000)   # 50,000 random long-only weights
ret = W @ mu                                                  # each portfolio's return
risk = np.sqrt(np.einsum("ij,jk,ik->i", W, cov, W))           # sqrt(w' cov w) for each row
sharpe = (ret - 0.02) / risk                                  # 2% risk-free rate
best, safest = sharpe.argmax(), risk.argmin()
mix = lambda w: {t: round(float(x), 2) for t, x in zip(prices.columns, w)}  # ticker -> weight
print(mix(W[best]), f"Sharpe {sharpe[best]:.3f}, return {ret[best]:.1%}, risk {risk[best]:.1%}")
print(mix(W[safest]), f"risk {risk[safest]:.1%}")

It prints the best mix (60% GLD, 29% NVDA, 7% GS, 4% XOM) with its Sharpe of about 1.61, return 34.7% and risk 20.3%, then the lowest-risk mix (59% GLD, 36% SPY, 4% XOM) at 13.6%, then the two nudged mixes of Table 5. dirichlet draws random weights that are never negative and always add up to 1. einsum("ij,jk,ik->i", ...) computes $w^\top \Sigma w$ for every row of W at once.

The script behind all the numbers

The frontier chart and tables come from examples/portfolio-optimisation/random_portfolios.py. To run it, save it next to the shared helpers market_data.py and chartstyle.py and the price file data/prices.csv, keeping the folders as in the links, then run uv run examples/portfolio-optimisation/random_portfolios.py.

Summary

Table 6 collects the formulas.

The series map puts the series’ ten ideas on one page.

References

• Philip J. Romero and Tucker Balch, What Hedge Funds Really Do: An Introduction to Portfolio Management (Business Expert Press, 2014), chapter 10: modern portfolio theory, the efficient frontier and the optimisation problem
• Yves Hilpisch, Python for Finance: Mastering Data-Driven Finance (2nd ed., O’Reilly, 2018), chapter 13: equations 13-1 and 13-2, the expected portfolio return and variance
• Harry Markowitz, “Portfolio Selection”, The Journal of Finance 7(1), 77–91, 1952
• Richard O. Michaud, “The Markowitz Optimization Enigma: Is ‘Optimized’ Optimal?”, Financial Analysts Journal 45(1), 31–42, 1989