Reinforcement learning: learning from reward, not from labels

The first series taught one shape of learning: a table of examples, each with its features and the right answer, and a model that learns to predict the answer for a new row. Every post in it, from linear regression to boosting, used that shape. This series starts with a different one. There is no right answer in the data at all. There is a situation, a choice, and afterwards a number that says how the choice turned out. That is reinforcement learning, and the first job is to see clearly what changed, because the data looks the same.

The same ten days, two questions

Take SPY’s daily prices for the first ten trading days of 2023. A supervised learner sees them as rows. Each row holds the last five daily returns as features and the next day’s return as the label (the right answer the model should predict):

The question those rows ask is “what will tomorrow’s return be?”. The right answer sits in the last column, the model is scored on how close it gets, and the rows are independent of each other: predicting 2023-01-04 badly does not change the row for 2023-01-05.

A reinforcement learner sees the same ten days as one episode. On the first morning it is in some situation (the state: say, those same five past returns, plus the fact that it holds nothing). It picks an action: buy SPY, or stay in cash. The day passes and it receives a reward: the day’s return on whatever it holds. Then it is in a new state, and chooses again. Nobody tells it whether buying was the right answer. It only ever sees the number.

Sutton and Barto draw exactly this loop (their Figure 3.1) and define the five words by it. The agent is the learner and decision maker. The environment is everything outside it, which responds to each action with a reward and a new state. The sequence the two produce together is state, action, reward, state, action, reward, and so on, and the agent’s one goal is to make the sum of the rewards as large as it can over the long run, not the next one.

The question changed, not the data. Supervised: given this row, what is the answer? Reinforcement: given this situation, what should I do, knowing that what I do changes the next situation and that the result arrives later as a number?

Five words, mapped to a trade

The words are easy to say and easy to mix up, so here is each one on the trading case, with the mistake that usually goes with it.

Two of these deserve a second look. The reward defines the goal (Sutton and Barto, section 3.2): it is the environment’s one-number verdict on what just happened, and the agent will do whatever earns most of it. Choose “today’s price change” and the agent learns to predict direction; choose “today’s change in account value after commissions” and it also learns that trading often is expensive. The learner optimises the reward you wrote down, not the goal you had in mind. The policy is the thing being learned. Sutton and Barto define it as a mapping from states to the probability of taking each action (section 3.5), which covers both the deterministic kind (in this state, always buy) and the stochastic kind (in this state, buy 80% of the time). Everything else in the system, including the value estimates that later posts build, exists only to improve the policy.

What a reinforcement learner is given, and what it is not

Sutton and Barto’s section 1.1 sets out the contrast with supervised learning, and Mitchell’s section 13.1 lists what makes the reinforcement setting harder. Side by side:

The “given” row is the heart of it. Supervised learning works when a knowledgeable supervisor can hand over correct, representative examples of the right behaviour. Sutton and Barto’s point is that in interactive problems this is often impossible: nobody can list the right trade for every market situation, and the situations where learning would help most are the ones nobody has labelled. So the agent has to learn from its own experience, which is what the reward signal is for.

The last two rows are consequences of the first. Because the action changes the next state, the agent partly chooses its own training data, which is why exploration becomes a problem that supervised learning never has. Mitchell calls out two more: the state the agent sees may be only part of the real situation (a camera cannot see behind the robot; five returns cannot see the news), and the reward for an action may arrive much later than the action itself. That last one is where the arithmetic gets interesting.

Delayed reward: who gets the credit?

Suppose the agent went long SPY on 2023-01-03 and held for ten days. Here is what it received, day by day, from the shared price file:

The agent made a decision every morning: buy on day 1, then “hold” on days 2 to 10. Which decisions were good? The reward of a day is a bad judge. Day 3’s decision to hold was followed by the worst day of the ten, a loss of $\bad{1.14\%}$, yet the days after it added up to $\good{3.63\%}$. Judged by what it led to, holding on day 3 was fine; judged by its own day, it looks like a mistake. This is Mitchell’s temporal credit assignment problem: the outcome arrives spread over many steps, and the learner has to work out which of the earlier actions deserve the credit or the blame.

Sutton and Barto’s answer is to judge each action by the whole sequence of rewards that follows it, which they call the return (equation 3.7, for an episode that ends at step T):

G, the return: what the decision at time t is answerable forR, the rewards observed afterwards

That is the last column of the table. The day-1 decision to buy is answerable for all ten days, $\key{G_1} = \good{3.98\%}$. The day-6 decision is answerable for the last five, $\key{G_6} = \good{2.53\%}$. The word “return” here is the reinforcement-learning word, the sum of future rewards; it is not the finance word for a price change, even though in this example each reward happens to be a daily price return.

The second panel is the picture to keep. Early decisions carry the weight of everything after them, and the last decision carries only its own day. Later posts in this series discount the later rewards so that a reward far away counts for less than one tomorrow, and estimate these sums without waiting for the episode to end. For now the point is simpler: in reinforcement learning the thing a decision is judged by is a sum over the future, and the learner has to build that judgement from rewards that arrive one at a time.

Episodes end; the market does not

The ten days above were an episode because I said so. Sutton and Barto (section 3.3) separate two kinds of task. An episodic task breaks naturally into pieces with a terminal state and a reset: a game of chess ends, a trip through a maze ends at the exit, and the next game starts fresh regardless of how the last one went. A continuing task has no natural end: a process controller runs as long as the plant does.

Trading is the second kind dressed up as the first. A market-making system runs whenever the exchange is open, and a portfolio carries its positions overnight, so there is no terminal state unless the designer draws one. Cutting a long history into year-long training episodes is useful, because it gives the learner many starts and many ends to learn from, but the cut is artificial: the holding at the end of one episode would have been the holding at the start of the next, and the agent is not told that. The post on states and rewards comes back to what the holding position does to the state. The word to carry forward is that an episode boundary is a design decision, and like the reward, it shapes what gets learned.

When this is the right tool, and when it is not

Reinforcement learning sounds more advanced than regression, and a research director who has read about it may want it for that reason alone. That is the wrong reason. Three questions tell you whether the problem is really the sequential kind:

Does the action change the next situation?a position held today is the position you start from tomorrow

Yes the problem is sequential, and the reward for today's choice includes what it makes possible tomorrow. A one-step predictor cannot see that.

No each decision stands alone, and a supervised model that predicts the outcome plus a rule that acts on it is simpler, cheaper to train and easier to check.

Can anyone write down the right answer for each situation?a label that a supervisor could supply

Yes use it. A dataset with correct labels is the strongest signal there is, and reinforcement learning throws it away in exchange for a weaker one.

No but you can score the outcome afterwards, which is exactly the reinforcement setting: a reward is a score, not an answer.

Can the learner afford to explore?trying an action it believes is worse, in order to learn

Yes in a simulator, or on paper trades, where a bad action costs nothing real.

No with live capital every exploratory trade is a real loss, so the learning has to happen in a model of the market first, and the model becomes the thing to worry about.

Sutton and Barto’s own framing of the boundary is that supervised learning is adequate for learning from a supervisor and inadequate for learning from interaction. Predicting tomorrow’s return is the first kind of problem. Deciding whether to enter, keep, reverse or exit a position, when each choice changes what the next choice costs, is the second. Many trading problems that look like the second are in fact the first with a position rule bolted on, and that version should be tried before the agent is built.

The code

The snippet reproduces the episode table. Variables:

import pandas as pd

url = "https://raw.githubusercontent.com/ankurtrapasiya/examples/main/data/prices.csv"
px = pd.read_csv(url, index_col="date", parse_dates=True)["SPY"]
r = (px / px.shift(1) - 1).dropna()

start = r.index.get_loc(pd.Timestamp("2023-01-03"))
ep = r.iloc[start:start + 10]            # reward each day = the day's return on a long position
to_go = ep[::-1].cumsum()[::-1]          # add up from the end: reward still to come after each day

for day, (date, reward) in enumerate(ep.items(), start=1):
    print(f"day {day:>2} {date.date()}  reward {100*reward:+.2f}%  still to come {100*to_go[date]:+.2f}%")
print(f"total {100*ep.sum():+.2f}%")

It prints the ten rows of the table and a total of +3.98%. The [::-1] idiom reverses a sequence, so ep[::-1].cumsum()[::-1] adds up from the last day backwards and flips the result back into date order.

Summary

References

• Sutton and Barto, Reinforcement Learning: An Introduction, 2nd edition (MIT Press, 2018). Section 1.1 for the definition of reinforcement learning, the two distinguishing features (trial-and-error search and delayed reward) and the contrast with supervised and unsupervised learning; section 1.3 for policy, reward signal, value function and model; section 3.1 and Figure 3.1 for the agent–environment loop; section 3.2 for the reward as the definition of the goal; section 3.3 for episodic and continuing tasks and the return, equation 3.7; section 3.5 for the policy as a mapping from states to action probabilities.
• Tom M. Mitchell, Machine Learning (McGraw-Hill, 1997). Section 13.1 for the learning task, the control policy, and the four things that set reinforcement learning apart from function approximation: delayed reward and temporal credit assignment, exploration, partially observable states and life-long learning.
• Kevin P. Murphy, Probabilistic Machine Learning: Advanced Topics (MIT Press, 2023). Section 35.1 for the trajectory of states, actions and rewards, and the split between model-free and model-based methods that later posts use.