Machine learning for quantitative finance in ten ideas

This is the map of the series: ten ideas, one short section each, in the order each one needs the one before it. Each section says what the idea is and why it matters, then links to the post that works it through with formulas, worked numbers and scikit-learn code. Read this page to see the whole shape, then go to the post for the idea you need.

The posts share one problem, so the ideas connect: from a stock’s own past prices, predict how volatile it will be next month (how much its price will swing). The inputs a model learns from, such as last month’s volatility, are called features.

Every number a model sees must have been knowable at the time. Every score needs a split, a metric and a baseline. Everything else is a choice of model.

1. The target decides the problem

Before choosing a model, decide what you are predicting. A number (next month’s volatility) makes it a regression problem; a label (“will next month be a stress month, far more volatile than usual, yes or no?”) makes it a classification problem, and the same inputs can serve both. Every target is part pattern, part noise nobody can predict, and the noise sets a floor no model gets under. On next-month volatility that floor is high: the best model in the series removes only about a third of the error you get by guessing the average every time. Knowing the floor stops you chasing a perfect model that cannot exist. Full post: What is machine learning?.

2. The data is part of the model

The algorithm gets the attention, but the choices made about the data (which rows to keep, how to fill gaps, how to treat extreme days, how to split into training and test) change the result as much as the algorithm does. A few extreme days can tilt a fitted line on their own; shuffling rows before splitting lets the model learn from days that come after the ones it is tested on, which it could never do live, so the test score looks better than the model really is. The post walks through each choice and what it does to the score. Full post: The data is part of the model.

3. A linear model is a weighted sum

The simplest model multiplies each input by a weight and adds them up. Learning means choosing the weights that make the misses on the training data as small as possible. Each weight reads as “one unit more of this input, everything else held fixed, moves the prediction by this much”, which makes the model easy to explain. Ridge and lasso are versions that keep the weights small, so the model does not lean too hard on any one input. Full post: Linear regression.

4. Every feature has a timestamp

For every input, ask: when would I have known this value, and could I compute it live? Leakage is an input that secretly contains the answer, usually because it uses information from the future. It is a common way a model looks brilliant and then fails in production. In the series, adding the market’s volatility over the very month being predicted made the model look clearly better than the honest version. That model could never run for real: at prediction time, that month has not happened yet. Full post: Every feature has a timestamp.

5. A tree asks one question at a time

A decision tree is a chain of yes/no questions, such as “was last month’s volatility above 25% a year?”, each one splitting the data into two groups, until each group gets one answer. It is easy to read and needs no scaling (putting the inputs on a common scale first). Its danger is that, left alone, it keeps asking questions until every training month sits in its own group: perfect on the past and much worse on new months. Stopping it after three questions in a row did much better on new months than letting it grow. Full post: Decision trees.

6. Nearest neighbours store everything

A completely different idea: learn nothing up front, keep every training row, and answer a new question by finding the most similar past rows and averaging what happened to them. Two things follow. “Similar” means close in distance, so the inputs must be put on the same scale first, or the input with the biggest numbers decides alone. And the model can only repeat what it has seen, so it cannot predict beyond the range of its past. The number of neighbours is the dial: few neighbours follow every quirk, many neighbours smooth them away. Full post: Nearest neighbours.

7. The gap between seen and unseen is the whole game

Making a model more flexible (more questions in a tree, fewer neighbours in nearest neighbours) always lowers its error on the data it trained on. Error on new data falls at first, then rises, because the model starts memorising noise. Compare the error on training data with the error on new data to diagnose any model: both high means it is too simple (underfit); low on training and much higher on new data means it memorised (overfit). More data helps the second case, not the first. Full post: Bias and variance.

8. A score needs a split, a metric and a baseline

A score means nothing on its own. You need to know which data it was measured on (data the model trained on always flatters it), what it measures, and what a trivial guess scores on the same data. For rare events, accuracy misleads: a model that always says “calm” is right most of the time and useless. The cut-off matters too: the model gives each month a probability of stress, and you choose the probability above which it raises an alarm. The same model caught about seven in ten real stress months with the cut-off at 50% and more than eight in ten with it at 20%, at the price of four times the false alarms. Choosing the cut-off is a business decision, not a modelling one. Full post: Scoring a model.

9. Averaging helps only when the errors differ

If several models make different mistakes, averaging them cancels part of each mistake; if they all make the same mistake, the average makes it too. Bagging trains many trees, each on a different random sample of the data, and averages them. Random forests go further and force the trees to differ (each split may only look at a few randomly chosen features), which is why they usually beat plain bagging. Bagging a straight line changes almost nothing: a line barely moves when the data changes a little, so every copy draws nearly the same line. Full post: Bagging.

10. Model choice is a systems decision

The models differ in more than accuracy: how long they take to train, how fast they answer, how much memory they need, whether they can be explained, and whether they can predict beyond what they have seen. In the series, the most volatile month in the training data was 138% a year, and the month after it came in at 52%. Asked about a month twice as wild as anything seen, the tree and neighbour models answered between 53% and 61%, close to that stored outcome, because they can only repeat what they have stored. The straight line followed its slope to 178%. Neither is “right”; it depends on which failure your system can live with. Choose for the system the model runs in, not for the top of a leaderboard. Full post: Choosing a model.