Regression vs classification: predicting a number or a label

In the first post we saw that machine learning means learning rules from past examples. The next thing to decide is what kind of answer you want. In trading, for example, predicting a stock’s future return (“how much?”) is a different job from deciding buy, sell or hold (“which one?”).

Two questions about the same month

From here on, this series uses one example: what will the market do next month, in risk terms? The data is five years of daily prices for five tickers (SPY, AAPL, JPM, XOM and GLD, 2019 to 2023), the same prices as the Quantitative Finance Series 1. For every trading day and ticker I compute seven facts that a trader would know at that close (the features): the last 1-, 5- and 20-day returns, the volatility of the last 5 and 20 days, the price against its 20-day average, and its Bollinger position (where the price sits between its Bollinger bands, lines drawn two standard deviations above and below the 20-day average: 0 at the average, +1 or −1 at a band). The thing to predict is the volatility of the next 20 trading days, about a month, annualised (scaled up to a yearly figure) the way the standard deviation post does.

A risk manager might ask two different questions:

Same data, same features. Different questions, so different models.

Why volatility and not the return itself? Because the return is close to unpredictable from price history alone (why prices are hard to beat explains why), while volatility clusters: a wild month tends to follow a wild month. Predicting risk is something these models can do, and it is what risk teams need.

Classification: “which one?”

The answer is a category (also called a class). Stress or calm. Fraud or normal. Approve, review or reject.

The table has 6,090 rows, one per ticker per trading day. I trained on the 3,685 rows from 2019 to 2021 and tested on the 2,405 rows from 2022 to 2023, in that order: in finance you always test on dates after the training data, never mixed in. The answer (the label) for each row is whether the month that followed was a stress month; that was true for 20.5% of the training rows.

I trained a classifier called logistic regression. (Yes, the name says “regression”, but it’s a classification model. The name is historical, and it confuses everyone.)

A good classifier doesn’t just say “stress” or “calm”. It gives a probability:

• SPY in late July 2023, after a calm month (7.7% volatility, up 4.4%): $\key{3\%}$ chance of a stress month ahead. The next month came in at 11%.
• SPY in May 2022, after a wild month (35% volatility, down 12%): $\key{49\%}$. The next month came in at 29%, just under the line.

A risk team can size positions off those probabilities.

The accuracy trap

On the test rows, the model’s accuracy (share of correct answers) was $\key{79.96\%}$. Sounds decent. But a “model” that simply says every month is calm is right $\key{77.38\%}$ of the time, because only 22.6% of test rows had a stress month ahead. It never catches a single one.

An Introduction to Statistical Learning (ISLP) measures a classifier by its error rate, the share of wrong answers (equation 2.8):

ŷ, the predictiony, the actual valuethe error

$$\risk{\text{error rate}} = \frac{1}{n} \sum_{i=1}^{n} I(\good{y_i} \neq \key{\hat{y}_i})$$

where $I(\cdot)$ is 1 when the prediction is wrong and 0 when it’s right. Here it’s $\risk{20.04\%}$, which is just 100% minus the accuracy.

There is a catch. The model only says “stress” when it’s more than 50% sure. At that cut-off it catches just 124 of the 544 test rows that really had a stress month ahead. When one answer is rarer than the other, accuracy and error rate are close to useless on their own. This is the same lesson as the backtest in the probability post: always compare with how common the event is. For rare events, look at the predicted probabilities, and at how many of the real cases the model catches.

Regression: “how much?”

The answer is a number on a scale. How volatile next month will be. How much a loan will lose. Tomorrow’s return.

Same rows, same seven features, but now the label is the number itself. I trained a linear regression model, which finds the best straight-line relationship between the inputs and the answer.

Its forecasts for the two days above: $\key{14.8\%}$ for the calm July 2023 day (actual 11.1%) and $\key{34.7\%}$ for the wild May 2022 day (actual 29.2%). Both on the right side, both a few points off.

How you measure a regression model

A regression answer is never exactly right, so you measure how far off it is on average. The standard measure, and the one ISLP uses (equation 2.5), is the mean squared error (MSE). ISLP writes it for the training rows; here it is computed the same way on the test rows, the ones the model has not seen:

$$\risk{\text{MSE}} = \frac{1}{n} \sum_{i=1}^{n} \left( \good{y_i} - \key{\hat{y}_i} \right)^2$$

Here the test MSE is $\risk{56.3}$. The unit is “percentage points squared”, which is hard to picture, so people usually take the square root to get back to points. That’s the root mean squared error:

$$\risk{\text{RMSE}} = \sqrt{\risk{\text{MSE}}} = \sqrt{56.3} = 7.5 \text{ points}$$

on months that average 22.8% volatility. A simpler alternative is the mean absolute error, $\sum |\good{y_i} - \key{\hat{y}_i}| / n$, the plain average size of a miss: $\risk{5.8}$ points here. Squaring makes MSE and RMSE punish one big miss more than several small ones, which suits problems where the big miss is the expensive one, like being set up for a 15% month that turns out to be a 45% one.

Both, side by side

On the left, each dot is a test row, sorted by predicted risk. Red dots are months that really were stressed, and most of them sit at the high end. On the right, each dot is a test row: the closer to the dashed line, the better the forecast.

The code

Both models are a few lines in scikit-learn. The pattern is always the same: create, fit on past data, predict on new data. The names:

Twelve made-up rows typed in by hand, so it runs as pasted (pip install scikit-learn):

from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Each row is one stock on one day: [volatility of the last 20 days (% a year),
#                                    return over the last 20 days (%)]
X_train = [[ 8,  3], [10,  2], [12, -1], [14,  4], [15, -2], [18,  1],
           [22, -5], [26, -8], [30, -3], [35, -12], [40, -9], [45, -15]]

# Two different answers for the same rows:
vol_train    = [9, 11, 12, 13, 17, 16, 24, 29, 27, 36, 38, 47]  # next month's volatility (% a year)
stress_train = [1 if v > 30 else 0 for v in vol_train]          # 1 = stress month (above 30%)

# Classification: "which one?" The answer is a category, stress or not.
clf = make_pipeline(StandardScaler(), LogisticRegression())
clf.fit(X_train, stress_train)

# Regression: "how much?" The answer is a number.
reg = LinearRegression()
reg.fit(X_train, vol_train)

X_new = [[9, 4], [33, -10]]               # a calm day and a wild day the models have not seen
print(clf.predict_proba(X_new)[:, 1])     # chance of a stress month, one per row
print(reg.predict(X_new))                 # next month's volatility, one per row

It prints about [0.01 0.52]: a 1% chance of a stress month after the calm day, 52% after the wild one. Then about [ 8.5 34.3]: the forecast volatility for each. predict_proba returns two columns per row, the chance of 0 and the chance of 1; [:, 1] takes the second column for every row.

The scaler (StandardScaler) puts every feature on the same footing before the logistic model sees them; on the real data the model fits badly without it, because daily returns are tiny numbers and volatilities are not.

The script behind the real numbers

The numbers in this post come from examples/regression-vs-classification/two_questions.py, which runs the same two models on all seven features of the blog’s five tickers. To run it yourself, clone the examples repo: its README has a three-command Docker setup, or run uv run regression-vs-classification/two_questions.py.

How to tell which one you need

You can often turn one into the other. A trading model can predict tomorrow’s return (regression), and then you buy if the prediction is above some level (classification). Many trading strategies work exactly this way; the stress-month label above is the regression target with a line drawn at 30%.

References

• James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning, with Applications in Python (Springer, 2023), section 2.1.5: regression versus classification; section 2.2: mean squared error (equation 2.5) and error rate (equation 2.8)
• scikit-learn: linear models