Probability basics

Probability sits under almost everything in machine learning and finance: judging whether a strategy’s result is luck, how a learner decides, how risk adds up. Kevin Murphy’s Probabilistic Machine Learning starts its maths with it in chapter 2. This post is the refresher I wanted: the five rules, each with a plain meaning, a finance example, the formula, and the numbers worked out.

The examples come from trade settlement. When two banks agree an OTC trade (a deal made directly between them, not on an exchange), cash and securities must later change hands on the agreed date. If that doesn’t happen on time, the trade fails to settle. Fails cost money, so banks care a lot about predicting them.

What a probability is

A probability is a number from 0 (never happens) to 1 (always happens). The everyday way to read it: out of many tries, what share ends in this outcome?

Say that out of 10,000 trades last month, 300 failed to settle:

a probability we worked outthe bad event: a fail, fraudthe risk signal we condition on: a mismatch, an alert

$$\begin{aligned} P(\bad{\text{fail}}) &= \frac{\text{trades that failed}}{\text{all trades}} \\ &= \frac{300}{10{,}000} \\ &= 0.03 \\ &= 3\% \end{aligned}$$

Read $P(\bad{\text{fail}})$ as “the probability of a fail”.

Rule 1: “not” (the complement)

Plain meaning: the chance something does not happen is everything else.

$$P(\text{not } A) = 1 - P(A)$$

This rule is most useful for “at least one” questions, which are hard to count directly.

Example: a desk sends 10 trades today. Each fails with probability 3%, independently of the others. What is the chance that at least one fails?

Counting “at least one” means one fail, two fails, three… That’s messy. The opposite is simple: no fails at all. Each trade settles with probability 0.97, so:

$$P(\bad{\text{no fails}}) = 0.97^{10} = 0.7374$$ $$P(\bad{\text{at least one fail}}) = 1 - 0.7374 = 0.2626$$

A 3% chance per trade becomes a $\key{26\%}$ chance that something fails today. Small risks add up fast when you repeat them.

Rule 2: “or” (the addition rule)

Plain meaning: add the two chances, then take away the overlap, because you counted it twice.

$$P(A \text{ or } B) = P(A) + P(B) - P(A \text{ and } B)$$

Example: 5% of trades have an SSI mismatch (the two banks’ standing settlement instructions, the “where to send the money” details, don’t match). 20% of trades are cross-border. 2% are both. What share needs extra checking because it is one or the other?

$$\begin{aligned} &P(\risk{\text{mismatch}} \text{ or } \risk{\text{cross-border}}) \\ &\quad = 0.05 + 0.20 - 0.02 = 0.23 \end{aligned}$$

Without subtracting the 2% overlap, you would say 25% and over-staff the checking team.

If two things can’t happen together (like “settled on time” and “failed”), the overlap is zero and you simply add.

Rule 3: “given” (conditional probability)

Plain meaning: once you know something, you only look at the cases where it’s true.

$$P(A \mid B) = \frac{P(A \text{ and } B)}{P(B)}$$

Read the bar $\mid$ as “given”. $P(\bad{\text{fail}} \mid \risk{\text{mismatch}})$ means “the chance of a fail, among trades with a mismatch”.

Example: here are the 10,000 trades split two ways:

$$P(\bad{\text{fail}} \mid \risk{\text{mismatch}}) = \frac{150}{500} = 30\%$$ $$P(\bad{\text{fail}} \mid \risk{\text{no mismatch}}) = \frac{150}{9{,}500} = 1.58\%$$

Overall, 3% of trades fail. But a trade with a mismatch fails $\key{30\%}$ of the time, about 19 times more often than a clean trade. That’s useful: fix mismatches first.

Watch the direction. These are not the same question:

$$\begin{aligned} P(\bad{\text{fail}} \mid \risk{\text{mismatch}}) &= \frac{150}{500} \\ &= 30\% \end{aligned}$$ $$\begin{aligned} P(\risk{\text{mismatch}} \mid \bad{\text{fail}}) &= \frac{150}{300} \\ &= 50\% \end{aligned}$$

“Most mismatches settle fine” and “half of all fails had a mismatch” are both true at the same time. Mixing them up is one of the most common mistakes in reading data.

Rule 4: “and” (the multiplication rule)

Plain meaning: the chance of both is the chance of the first, times the chance of the second once the first has happened. Bishop and Bishop’s Deep Learning: Foundations and Concepts calls this the product rule (equation 2.9).

$$P(A \text{ and } B) = P(B) \times P(A \mid B)$$

Check it on the table: $P(\risk{\text{mismatch}}) = 500/10{,}000 = 5\%$, and $P(\bad{\text{fail}} \mid \risk{\text{mismatch}}) = 30\%$, so

$$\begin{aligned} &P(\bad{\text{fail}} \text{ and } \risk{\text{mismatch}}) \\ &\quad = 0.05 \times 0.30 = 0.015 = 1.5\% \end{aligned}$$

which matches the table: 150 out of 10,000.

Independence is the special case where knowing one thing tells you nothing about the other. Then $P(A \mid B) = P(A)$, and the rule becomes a plain multiplication, but only for independent events:

$$P(A \text{ and } B) = P(A) \times P(B)$$

I used this in the “at least one fail” example. In real settlement, fails are often not independent. If a counterparty has an operations problem, or a market is closed, many trades fail together. Multiplying probabilities that are actually linked makes risk look much smaller than it is.

Rule 5: Bayes’ rule (turning “given” around)

Plain meaning: Bayes’ rule turns “how likely is the evidence if it’s true” into the question you actually care about: “how likely is it true, now that I’ve seen the evidence”. It’s equation 2.10 in Bishop and Bishop:

$$P(A \mid B) = \frac{P(B \mid A) \times P(A)}{P(B)}$$

Example: a fraud alert. A card fraud model is “99% accurate”:

• It flags $\risk{99\%}$ of real fraud: $P(\risk{\text{alert}} \mid \bad{\text{fraud}}) = 0.99$
• It wrongly flags $\risk{1\%}$ of normal payments: $P(\risk{\text{alert}} \mid \text{normal}) = 0.01$
• Fraud is rare, $\bad{0.1\%}$ of payments: $P(\bad{\text{fraud}}) = 0.001$

An alert fires. How likely is it to be real fraud? Most people guess about 99%.

First, how often does an alert fire at all? Alerts come from fraud or from normal payments (this step is called the law of total probability; Bishop and Bishop derive it from the sum rule, equations 2.8 and 2.11):

$$\begin{aligned} P(\risk{\text{alert}}) &= 0.99 \times 0.001 + 0.01 \times 0.999 \\ &= 0.00099 + 0.00999 \\ &= 0.01098 \end{aligned}$$

Then Bayes:

$$\begin{aligned} P(\bad{\text{fraud}} \mid \risk{\text{alert}}) &= \frac{0.99 \times 0.001}{0.01098} \\ &= \frac{0.00099}{0.01098} \approx 0.09 \end{aligned}$$

Only about $\key{9\%}$ of alerts are real fraud. It’s easier to see with counts. Out of 100,000 payments:

• 100 are fraud, and the model flags 99 of them.
• 99,900 are normal, and the model wrongly flags 1% of them: 999.

The model is very good. The alerts are still mostly false, because normal payments outnumber fraud a thousand to one, and 1% of a huge number is still a big number.

Bishop and Bishop work through the same trap with a medical screening test (equations 2.20 to 2.22): a cancer that affects 1% of people, a test that catches 90% of cases and wrongly flags 3% of healthy people. A positive result means only about a $\key{23\%}$ chance of cancer.

This mistake has a name: base rate neglect. People look at how accurate the test is and forget how rare the thing is. It’s why fraud teams care so much about the false alarm rate, and why one alert alone is rarely enough to block a customer.

The code

The Bayes calculation in a few lines, plus a simulation of 5 million payments that checks the answer by brute force:

import numpy as np

prior, hit_rate, false_alarm = 0.001, 0.99, 0.01

# Bayes' rule
p_alert = hit_rate * prior + false_alarm * (1 - prior)
print(hit_rate * prior / p_alert)          # 0.0902

# Check it by simulation: make 5 million payments and count
rng = np.random.default_rng(0)
is_fraud = rng.random(5_000_000) < prior
alert = np.where(is_fraud,
                 rng.random(5_000_000) < hit_rate,
                 rng.random(5_000_000) < false_alarm)
print(is_fraud[alert].mean())              # 0.0908, close to 0.0902

Summary

Next

In the next post in this series: expected value, the average result you get when you repeat a risky decision many times, and how banks use it to price the risk of a loan.

References

• Christopher M. Bishop and Hugh Bishop, Deep Learning: Foundations and Concepts (Springer, 2024), chapter 2, section 2.1: sum and product rules (equations 2.8, 2.9), Bayes’ theorem (2.10, 2.11), the screening-test example (2.20 to 2.22)
• Kevin P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, 2022), chapter 2