States and rewards: what you tell the learner decides what it learns

Every post in this series so far has used a state with two values. In Q-learning: one update rule, repeated it was the holding, cash or long; in Dyna: learn a model of the world, then practise in it it was what SPY did yesterday, up or down. Two values kept the arithmetic small enough to do by hand. A real trading learner looks at indicators, and an indicator is a number on a continuous scale: the Bollinger position on the evening of 2022-12-30 was −0.3849, and the next evening it will be something else that has never appeared before. A Q-table has one row per state, so before it can learn anything somebody has to decide which readings count as the same state. That decision, and the matching decision about what the reward is, shape what the learner can learn more than the update rule does. This post works through both on SPY from 2020 to October 2025 and on the ten January 2023 days the series has used all along.

A Q-table needs a list of states

Mitchell names the limit directly. The Q-learning he describes keeps “an explicit lookup table, with a distinct table entry for every distinct input value”, so it does “a kind of rote learning” and makes no attempt to estimate an unseen state from similar ones (section 13.6). A table of indicator readings would have a new row almost every day, and each row would be updated once. The usual move is to cut each indicator’s range into a few pieces, called bins, and treat every reading that falls in the same piece as the same state. Sutton and Barto call this state aggregation: “states are grouped together, with one estimated value … for each group” (section 9.3). Whatever the learner learns about a group, it applies to every reading inside it.

Take three indicators of SPY from the indicators post: the Bollinger position (where the close sits between the bands, −1 at the lower band and +1 at the upper), the 20-day return, and the 20-day volatility (the standard deviation of the last 20 daily returns). Cut each one at its quartiles, the readings that a quarter, a half and three quarters of the 1,447 days fall below. Three thresholds make four bins, numbered 0 to 3, and each bin holds a quarter of the days. A reading’s bin is simply how many of the three thresholds it is above. Table 1 has the thresholds and the readings on the evening of 2022-12-30, the evening before day 1 of the January 2023 episode:

The Bollinger reading of −0.3849 is below all three thresholds, so it is in bin 0, the lowest quarter. The 20-day return is also in its lowest quarter, and the volatility is above two thresholds, so bin 2. In words: SPY closed in the lower part of its bands, after a falling month, in a fairly nervous market. Figure 1’s first panel shows the Bollinger position over the six years with the three thresholds drawn across it; the dot is this evening.

Three bin numbers become one state label the way digits become a number. Read them in a fixed order and write them side by side, 0, 0, 2 gives “002”. If the table wants a row number instead of a string, weight each digit by a power of the number of bins, as the hundreds, tens and ones of an ordinary number are weighted by powers of ten:

s, the state label, one row of the Q-table

Here $b_1$, $b_2$ and $b_3$ are the three bin numbers, each 0 to 3. For 2022-12-30, $\key{s} = 16 \times 0 + 4 \times 0 + 2 = \key{2}$, and the labels run from 0 (all three in bin 0) to 63 (all three in bin 3). Nothing about the label is clever. It is a row number, and the only thing it must do is give two evenings the same row exactly when they fall in the same three bins.

Note. The thresholds above use all six years, which is fine for counting, as here. A learner trained on 2020 to 2023 has to take its thresholds from those years only; thresholds computed on the full history would tell it where the future’s readings fall, which every feature has a timestamp covers in detail.

Too few bins merge situations, too many leave states empty

The number of possible labels is the number of bins multiplied by itself once per indicator:

N, the number of possible states

with $b$ bins per indicator and $d$ indicators. Three indicators at four bins give $\key{N} = 4^3 = \key{64}$. The days, however, are fixed: 1,447 of them, whatever the bins are. Table 2 counts, for each choice of $b$, how many states the bins create, how many at least one day lands in, and how many receive ten days or more:

Two things go wrong, one at each end.

Too few bins. At two bins per indicator every state is visited more than a hundred times, which looks healthy, but each bin is now half of all the readings. The lower half of the Bollinger position runs from $-1.76$, the close of 2025-04-04 during the April 2025 sell-off, to $+0.42$, a close a little above the 20-day average. Those two evenings get the same label on that axis. If the right action after a crash is different from the right action on a quiet day, the learner cannot find it, because it never sees the difference: both evenings update the same row. The MDPs post stated this as a demand on the state, and here it is in numbers. No learning rate, exploration rate or amount of training fixes it, because the information is not in the label.

Too many bins. At ten bins per indicator the bins create 1,000 states, the data lands in 506 of them, and only 10 are visited ten times or more. The median visited state has two days in it, and 1,305 of the 1,447 days sit in states seen fewer than ten times. Each of those rows has been updated once or twice from one or two noisy daily returns. Mitchell’s convergence proof for the table holds “only if every possible state-action pair is visited (infinitely often!)”, which he calls unrealistic in large spaces (section 13.6); 494 states with no visit at all keep whatever starting value they were given. Sutton and Barto put the general problem in the introduction to their Part II: “The problem with large state spaces is not just the memory needed for large tables, but the time and data needed to fill them accurately.” Figure 1’s second panel shows the gap opening: the states created climb by a factor of $b^3$, and the well-visited ones turn down after five bins.

Adding an indicator multiplies the count by $b$, because $d$ is the exponent. Seven features at ten bins each would make 10,000,000 states for 1,447 days. That is the point at which a table stops being the right tool, and Sutton and Barto’s Part II replaces it with a function that generalises from visited states to similar unvisited ones. Inside a table, the levers are fewer indicators, fewer bins, and a count like Table 2 before any learning starts.

Holdings belong in the state

The same evening means a different thing to two different agents. An agent in cash that wants to be long has to buy, and buying costs money. An agent already long gets the same exposure by doing nothing. Ritter’s paper says the state “will include the agent’s current position, which is clearly an observable that is an important determinant of the next action” (section 3.1), and his worked example sets the state to the price and the position coming into the period, so the state space is every price paired with every position (section 4).

Day 1 of the January 2023 episode shows it in four numbers. SPY returned $\good{-0.42\%}$ that day. Charge a cost of $\bad{0.10\%}$ of the trade whenever the holding changes; that is a round number for the example, and What a trade really costs breaks a real cost into spread, commission and impact. Table 3 gives the day’s reward for each holding and action:

“Go long” earns $\key{-0.52}$ from cash and $\key{-0.42}$ from long. “Go cash” earns $\key{0.00}$ from cash and $\key{-0.10}$ from long. If the state is only the indicator label “002”, both agents update the same two cells, and each cell ends up a blend of two different situations. With costs the blend matters: a cell that cannot tell “buy” from “hold” cannot learn that buying is the expensive one. Adding the holding as one more digit fixes it. With cash as 0 and long as 1, the label becomes $16\,b_1 + 4\,b_2 + b_3 + 64 \times \text{holding}$, so “002” is row 2 in cash and row 66 when long, and the state count doubles from 64 to 128. Unlike an indicator, the holding needs no bins: it already has exactly two values.

Four rewards for the same ten days

The reward is the second thing the designer chooses, and Sutton and Barto put the rule plainly in section 3.2: “The agent always learns to maximize its reward. If we want it to do something for us, we must provide rewards to it in such a way that in maximizing them the agent will also achieve our goals.” Ritter’s reward starts from the change in portfolio value over one period, net of costs (his equation 6, before the risk term he adds in section 3.5):

R, the reward: the change in portfolio value over the periodr, the asset's return over the periodc, the trading cost paid

where $h_{t-1}$ is the holding in dollars over the period. In percent of the account, with the holding 1 when long and 0 in cash, that is the day’s return if long, zero if in cash, minus $\bad{0.10\%}$ on a day the holding changes.

To compare rewards, the actions have to be fixed, so take a simple rule built from the Dyna post’s state: long after a down day, cash after an up day, starting in cash. On the ten days it buys before day 1, sells before day 3, buys before day 4 and so on, six trades in all. Table 4 scores each day four ways: SPY’s price change whatever the agent holds; the account value change without costs; the realised gain, zero until a position is sold and then the whole gain of that position, booked on the step that sells; and the gain after costs, Ritter’s $\key{R_t}$.

Same ten days, same six trades, four totals, and each reward teaches a different habit.

Price change totals $\key{+3.98}$, the episode’s return from the MDPs post, but it is the same number whichever action the agent takes. On day 3 the agent sat in cash and was still scored −1.14. A reward that does not depend on the action carries no information about which action is better, so a learner trained on it learns nothing about trading. It would be scoring the market, not itself.

Account value change totals $\key{+3.35}$, the four long days added up. It depends on the action and arrives every day, so the learner hears about a bad day the evening it happens. It ignores costs, so a rule that trades every day and one that trades twice a year look the same when their days add up the same.

Realised gain totals $\key{+3.34}$, nearly the same (the gap is compounding inside the two-day position), but it is zero on seven of the ten days. Ritter’s accounting calls a gain “unrealized until we trade again and move the profit into the cash term, at which point it is realized” (section 2.1), and a reward built on that line only speaks when the agent sells. The habit it teaches is about losses. Selling a losing position books the loss as a reward now; holding it books nothing now, and with the discount factor from the MDPs post a loss booked later counts for less. A loss of $\bad{1.14\%}$ booked today counts in full; booked ten steps later at $\gamma = 0.9$, it counts as $\key{-0.40}$. The learner is paid to hold losers.

Gain after costs totals $\key{+2.75}$: the account value change minus six trades at $\bad{0.10\%}$. It is the only one of the four that tells the learner what the account actually did, and the only one in which trading less is worth something. Ritter goes one step further and subtracts a risk term from it, so the reward becomes the change in value minus a penalty on its square (equations 13 and 15); with that penalty set to zero his agent is “risk-neutral”, and he reports that maximising profit alone gave a much lower out-of-sample Sharpe ratio, return per unit of risk, than maximising the risk-adjusted version (section 4). The reward is where the designer’s attitude to risk enters, or fails to.

A bonus teaches the learner to collect the bonus

When a learner is slow, the tempting fix is to add a little extra reward for behaviour the designer believes leads to the goal. Sutton and Barto’s chess example in section 3.2 is the warning: a chess agent “should be rewarded only for actually winning, not for achieving subgoals such as taking its opponent’s pieces”, because it “might find a way to take the opponent’s pieces even at the cost of losing the game”. They close the section with the line the whole post rests on: “The reward signal is your way of communicating to the robot what you want it to achieve, not how you want it achieved.”

A trading version. A designer worries that the learner sits in cash and never tries anything, so they add a bonus of $0.15\%$ to the reward for every trade, meant to encourage it to explore. Each trade costs $\bad{0.10\%}$, so in the reward each trade is now worth $0.15 - 0.10 = \key{+0.05}$, whatever the market does. On the ten days of Table 4 the six trades lift the shaped total from $\key{+2.75}$ to $\key{+3.65}$, higher than the $\key{+3.35}$ the account would have made with no costs at all. A learner that flips its position every day trades ten times, loses $\bad{1.00\%}$ of the account to costs, and collects $1.50\%$ of bonus for it. The learner has not misbehaved. It found the reward it was given, and the reward says trading is good.

Sutton and Barto’s later chapter on designing reward signals says the same thing in general terms: adding “well-intentioned supplemental rewards may lead the agent to behave very differently from what is intended”, and agents “can discover unexpected ways to make their environments deliver reward” (section 17.4). Their suggested fix is to leave the reward alone and put the designer’s guess into the starting values instead: it changes where learning starts, not what the reward asks for, though the book adds that it is not guaranteed to speed learning up.

The learner optimises the reward, not the intention behind it. If a number goes into the reward, expect the learner to chase that number, including the parts that were only meant to help.

Traps

Table 5 lists the usual misreadings.

The code

The snippet reproduces the bins of 2022-12-30 in Table 1, the four-bin row of Table 2 and Table 3. Its variables are in Table 6:

import pandas as pd

url = "https://raw.githubusercontent.com/ankurtrapasiya/examples/main/data/prices.csv"
px = pd.read_csv(url, index_col="date", parse_dates=True)["SPY"]
f = pd.DataFrame({
    "bollinger": (px - px.rolling(20).mean()) / (2 * px.rolling(20).std()),  # -1 lower band, +1 upper
    "ret_20d": px / px.shift(20) - 1,                                         # return over 20 days
    "vol_20d": px.pct_change().rolling(20).std(),                             # daily volatility, 20 days
}).dropna()

q = f.quantile([0.25, 0.5, 0.75])         # three thresholds per indicator: four bins
x = f.loc["2022-12-30"]                    # the evening before day 1 of the episode
bins = [int((x[c] > q[c]).sum()) for c in f.columns]   # thresholds the value is above: bin 0 to 3
print("bins:", bins, "label:", "".join(map(str, bins)))

label = sum(pd.qcut(f[c], 4, labels=False) * 4 ** (2 - k) for k, c in enumerate(f.columns))
visits = label.value_counts()              # days that landed in each state that occurred
print(f"{4 ** 3} states possible, {len(visits)} visited, {(visits >= 10).sum()} visited ten times or more")

cost, ret = 0.10, -0.42                    # % of the trade; SPY's return on day 1, in %
print("in cash:      go long", round(ret - cost, 2), " go cash", 0.0)
print("already long: go long", ret, " go cash", -cost)

It prints bins [0, 0, 2] and the label 002, then 64 states possible, 56 visited and 43 visited ten times or more, then the four rewards of Table 3: −0.52 and 0.0 from cash, −0.42 and −0.1 from long. pd.qcut(..., 4, labels=False) is the whole-column version of the threshold count above it: it cuts a column at its quartiles and returns each row’s bin number.

Summary

Table 7 is the post in one line per idea.

References

• Sutton and Barto, Reinforcement Learning: An Introduction, 2nd edition (MIT Press, 2018). Section 3.2 for the reward hypothesis, the agent always maximising its reward, the chess subgoal example and the reward as what to achieve, not how; the introduction to Part II for large state spaces needing time and data, not only memory; section 9.3 for state aggregation; section 17.4 for designing reward signals, the risk of supplemental rewards, and putting prior knowledge into initial values instead.
• Tom M. Mitchell, Machine Learning (McGraw-Hill, 1997). Section 13.1 for partially observable states; section 13.6 for the explicit lookup table, rote learning, and convergence only if every state–action pair is visited infinitely often.
• Gordon Ritter, Machine Learning for Trading (2017, SSRN 3015609). Section 2.1 for realised and unrealised profit and loss; section 2.2 for the change in portfolio value net of costs (equation 6); section 3.1 for the state including the agent’s current position; section 3.5 for the risk-adjusted reward (equations 13 and 15); section 4 for the state as price and position, and the risk-neutral reward giving a lower Sharpe ratio.