Regression vs classification: predicting a number or a label

In the first post we saw that machine learning means learning rules from past examples. The next thing to decide is what kind of answer you want. In trading, for example, predicting a stock’s future return (“how much?”) is a different job from deciding buy, sell or hold (“which one?”).

Two questions about the same month

From here on, this series uses one example: what will the market do next month, in risk terms? The data is five years of daily prices for five tickers (SPY, AAPL, JPM, XOM and GLD, 2019 to 2023), the same prices as the finance stats series. For every trading day and ticker I compute seven facts that a trader would know at that close (the features): the last 1-, 5- and 20-day returns, the volatility of the last 5 and 20 days, the price against its 20-day average, and its Bollinger position. The thing to predict is the volatility of the next 20 trading days, about a month, annualised the way the standard deviation post does.

A risk manager might ask two different questions:

Same data, same features. Different questions, so different models.

Why volatility and not the return itself? Because the return is close to unpredictable from price history alone (the models in this series score about zero on it), while volatility clusters: a wild month tends to follow a wild month. Predicting risk is something these models can do, and it is what risk teams need.

Classification: “which one?”

The answer is a category (also called a class). Stress or calm. Fraud or normal. Approve, review or reject.

The table has 6,090 rows (ticker-days). I trained on the 3,685 rows from 2019 to 2021 and tested on the 2,405 rows from 2022 to 2023, in that order: in finance you always test on dates after the training data, never mixed in. The answer (the label): 20.5% of the training months turned out to be stress months.

I trained a classifier called logistic regression. (Yes, the name says “regression”, but it’s a classification model. The name is historical, and it confuses everyone.)

A good classifier doesn’t just say “stress” or “calm”. It gives a probability:

• SPY in late July 2023, after a calm month (7.7% volatility, up 4.4%): $\key{3\%}$ chance of a stress month ahead. The next month came in at 11%.
• SPY in May 2022, after a wild month (35% volatility, down 12%): $\key{49\%}$. The next month came in at 29%, just under the line.

A risk team can size positions off those probabilities.

The accuracy trap

On the test rows, the model’s accuracy (share of correct answers) was $\key{79.96\%}$. Sounds decent. But a “model” that simply says every month is calm is right $\key{77.38\%}$ of the time, because only 22.6% of test months were stress months. It never catches a single one.

An Introduction to Statistical Learning (ISLP) measures a classifier by its error rate, the share of wrong answers (equation 2.8):

ŷ, the predictiony, the actual valuethe error

$$\risk{\text{error rate}} = \frac{1}{n} \sum_{i=1}^{n} I(\good{y_i} \neq \key{\hat{y}_i})$$

where $I(\cdot)$ is 1 when the prediction is wrong and 0 when it’s right. Here it’s $\risk{20.04\%}$, which is just 100% minus the accuracy.

There is a catch. The model only says “stress” when it’s more than 50% sure. At that cut-off it catches just 124 of the 544 months that really were stressed. When one answer is rarer than the other, accuracy and error rate are close to useless on their own. This is the same lesson as the backtest in the probability post: always compare with how common the event is. For rare events, look at the predicted probabilities, and at how many of the real cases the model catches.

Regression: “how much?”

The answer is a number on a scale. How volatile next month will be. How much a loan will lose. Tomorrow’s return.

Same rows, same seven features, but now the label is the number itself. I trained a linear regression model, which finds the best straight-line relationship between the inputs and the answer.

Its forecasts for the two days above: $\key{14.8\%}$ for the calm July 2023 day (actual 11.1%) and $\key{34.7\%}$ for the wild May 2022 day (actual 29.2%). Both on the right side, both a few points off.

How you measure a regression model

A regression answer is never exactly right, so you measure how far off it is on average. The standard measure, and the one ISLP uses (equation 2.5), is the mean squared error (MSE):

$$\risk{\text{MSE}} = \frac{1}{n} \sum_{i=1}^{n} \left( \good{y_i} - \key{\hat{y}_i} \right)^2$$

Here the test MSE is $\risk{56.3}$. The unit is “percentage points squared”, which is hard to picture, so people usually take the square root to get back to points. That’s the root mean squared error:

$$\risk{\text{RMSE}} = \sqrt{\risk{\text{MSE}}} = \sqrt{56.3} = 7.5 \text{ points}$$

on months that average 22.8% volatility. A simpler alternative is the mean absolute error, $\sum |\good{y_i} - \key{\hat{y}_i}| / n$, the plain average size of a miss: $\risk{5.8}$ points here. Squaring makes MSE and RMSE punish one big miss more than several small ones, which suits problems where the big miss is the expensive one, like being set up for a 15% month that turns out to be a 45% one.

Both, side by side

On the left, each dot is a test row, sorted by predicted risk. Red dots are months that really were stressed, and most of them sit at the high end. On the right, each dot is a test row: the closer to the dashed line, the better the forecast.

The code

Both models are a few lines in scikit-learn. The pattern is always the same: create, fit on past data, predict on new data.

from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Classification: will next month be a stress month?   y = 1 (yes) or 0 (no)
clf = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
clf.predict_proba(X_new)[:, 1]            # probability of a stress month, for each row

# Regression: how volatile will next month be?          y = next month's volatility, % a year
reg = LinearRegression().fit(X_train, vol_train)
reg.predict(X_new)                        # e.g. 34.7 for SPY in May 2022

(The scaler puts every feature on the same footing before the logistic model sees them; the model fits badly without it, because daily returns are tiny numbers and volatilities are not.)

How to tell which one you need

You can often turn one into the other. A trading model can predict tomorrow’s return (regression), and then you buy if the prediction is above some level (classification). Many trading strategies work exactly this way; the stress-month label above is the regression target with a line drawn at 30%.

Next

In the next post in this series: linear regression, properly. How the “best straight line” is found, what the numbers it learns actually mean, and the maths behind it.

References

• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 2.1.5 (regression versus classification) and section 2.2: mean squared error (equation 2.5) and error rate (equation 2.8)
• scikit-learn: linear models