Bayes' rule

When you reach for it

Information often arrives the wrong way round. A factory knows how its inspection camera behaves on a faulty circuit board and on a good one: it knows P(\text{flagged} \mid \text{faulty}) and P(\text{flagged} \mid \text{good}). The person holding a flagged board wants the other direction, P(\text{faulty} \mid \text{flagged}): given what the camera said, what is the board really like?

These two conditional probabilities are different numbers, and they can be wildly different. Bayes’ rule is the move that turns one into the other. Reach for it whenever

• the situation splits into cases (faulty or good, which supplier, which coin), and you know the chance of each case,
• you know how likely some piece of evidence is within each case, and
• the evidence has now been seen, and you want the chance of a case given the evidence.

The question “given the result, what was the cause?” is the signal. It covers rare faults and the false alarms they bring with them, reading back from an effect to the supplier that produced it, and updating a judgment when a second piece of evidence arrives.

The procedure

The method rests on three things already known: the formula for a conditional probability, the general multiplication rule and the law of total probability. Nothing new is assumed; Bayes’ rule is these three used in the right order.

Draw the tree in the direction the information comes. The first fan of the tree is the partition into cases, labeled with their chances P(B_i). The second fan, from each case, is the evidence, labeled with the chances given that case, P(A \mid B_i). This is the order the numbers are known in, so every branch gets a number that was actually given. The branch you want, a case given the evidence, appears nowhere on this tree, which is exactly why a calculation is needed.

Multiply along each path. The general multiplication rule says P(B_i \text{ and } A) = P(B_i)\,P(A \mid B_i), so the product along a path is the probability of that whole path. The paths do not overlap and together they cover everything, so their products add to 1.

Add the paths that match the evidence. Once the evidence A is seen, only the paths ending in A are still possible. Their products add to P(A), by the law of total probability. This total is the size of the world that is left.

Divide the wanted path by that total. The formula for a conditional probability says \displaystyle P(B_k \mid A) = \frac{P(B_k \text{ and } A)}{P(A)}. The top is the one path through B_k that ends in A; the bottom is the total just found. So the answer is the wanted path’s share of all the paths that match the evidence.

With two cases, B and “not B”, this is

where

With a partition B_1, B_2, \ldots, B_n into any number of cases,

That is Bayes’ rule. On the tree it reads: the wanted path, divided by the sum of all the paths ending in the evidence.

The same move with counts. Probabilities can be traded for counts by imagining a round number of items, say 10{,}000, and splitting them along the tree. Each branch takes its share of the items that reach it, so the counts at the leaves are the path products times 10{,}000. The answer is then “how many of the items showing the evidence are in the wanted case”, a plain fraction of two whole numbers. Counts and probabilities give the same answer, because the 10{,}000 cancels top and bottom.

Worked examples

Boundaries

The evidence must be possible. The division is by P(A), so the rule needs P(A) > 0. If no path ends in the evidence, there is nothing to condition on.

The cases must form a partition. As in the law of total probability, the cases must not overlap and must cover every outcome, or the bottom of the fraction counts some paths twice or leaves some out. The checks are that the case chances add to 1 and that the answers P(B_i \mid A) add to 1 as well.

The base rate is needed. The rates P(A \mid B_i) alone cannot give P(B_i \mid A). The camera’s 95\% and 8\% gave answers from 0.012 to 0.922 in Example 5, depending on the share of faulty boards. Without P(B_i), the reverse probability is not determined.

The base rate must fit the item. The 1\% in Example 1 is the share of faulty boards among boards chosen at random. A board picked out because it smells burnt is not a random board, and its chance of being faulty before the camera looks is no longer 0.01, so the answer 0.107 does not apply to it.

Updating twice needs independence given the case. Example 4 multiplied the X-ray’s rates onto the camera’s paths, which is right only if, for a board of each kind, the X-ray’s result does not depend on the camera’s. Two runs of the same camera that misread the same speck of dust are not independent in this way, and treating them as two separate tests overstates how much the second flag tells you.

Where this goes next

Bayes’ rule is the turning point of conditional probability: it is how a chance known one way round is read the other way. Its denominator is the law of total probability, and its division is the formula for a conditional probability, so the three belong together.

The habit it teaches, starting from a base rate and updating it as evidence arrives, as in Example 4, is the foundation of a whole way of doing statistics, called Bayesian statistics, in which every unknown carries a probability that each new observation adjusts. The same reversal sits behind judging any detector or screening check: how often it is right about the items that have the condition is one question, and how often an alarm is real is another, and only the base rate links them. When a test result turns out to change nothing, as in Exercise 18, the evidence and the case are independent events.

Exercises

Reversing on a tree

1. P(B) = 0.3, P(A \mid B) = 0.6 and P(A \mid \text{not } B) = 0.2. Find P(B \mid A).
2. With the numbers of Exercise 1, find P(B \mid \text{not } A).
3. A second-hand bookshop’s stock is 70\% paperbacks and 30\% hardbacks. Of the paperbacks, 10\% have a torn page; of the hardbacks, 4\%. A book picked at random has a torn page. Find the probability that it is a hardback.
4. On 20\% of nights a fox visits a garden. A motion-sensitive camera in the garden is set off on 80\% of the nights the fox visits and, by wind and cats, on 15\% of the nights it does not. Find the probability that the fox visited on a night the camera was set off, and on a night it was not.

Three or more cases

5. A phone repair shop buys 50\% of its replacement screens from supplier A, 30\% from B and 20\% from C. Of the screens from A, 1\% are faulty; from B, 2\%; from C, 3\%. A screen is faulty. Find the probability that it came from each supplier.
6. Bag 1 holds 2 red and 3 blue counters, bag 2 holds 4 red and 1 blue, and bag 3 holds 5 red. A fair die is rolled: on 1, 2 or 3 a counter is drawn from bag 1, on 4 or 5 from bag 2, and on 6 from bag 3. The counter is red. Find the probability that it came from bag 2.
7. P(A) = 0.25, P(B) = 0.4 and P(A \mid B) = 0.5. Find P(B \mid A) and P(A \mid \text{not } B).

Rare events and false alarms

8. Of the payments made with a bank’s cards, 0.2\% are fraudulent. The bank’s fraud check flags 99\% of fraudulent payments and 2\% of genuine ones. Find the probability that a flagged payment is fraudulent.
9. A kit tests well water for a contaminant. Of the wells in a region, 5\% are contaminated. The kit gives a positive result for 90\% of contaminated wells and for 10\% of clean ones. Picture 1000 wells: make a two-way table of contaminated or clean against positive or negative, and use it to find P(\text{contaminated} \mid \text{positive}) and P(\text{clean} \mid \text{negative}).
10. A well from Exercise 9 tests positive and is tested again with a second kit of the same kind, whose result, for a well of each kind, is independent of the first. The second result is also positive. Find the probability that the well is contaminated.
11. For the kit of Exercise 9, what share of wells would have to be contaminated for a positive result to mean a \displaystyle \frac{1}{2} chance of contamination?

Fractions and repeated evidence

12. A drawer holds four coins: three fair and one with heads on both sides. A coin is taken at random and tossed three times, showing heads each time. Find the probability that it is the two-headed coin.
13. Box A holds 6 bulbs, 2 of which do not work. Box B holds 4 bulbs, 1 of which does not work. A fair coin chooses a box and a bulb is taken from it at random. The bulb does not work. Find the probability that it came from box A.
14. On a test where every question has 4 options, a student knows the answer to 60\% of the questions and picks an option at random on the rest. A question was answered correctly. Find the probability that the student knew the answer.

Reasoning

15. Use the table of Example 2 to explain, in terms of the flagged boards, why P(\text{faulty} \mid \text{flagged}) is so much smaller than P(\text{flagged} \mid \text{faulty}).
16. Show that P(A)\,P(B \mid A) = P(B)\,P(A \mid B) whenever P(A) > 0 and P(B) > 0, and use it to derive the two-case form of Bayes’ rule.
17. A check never flags an item without the condition: P(A \mid \text{not } B) = 0, while P(A \mid B) > 0 and P(B) > 0. Show that P(B \mid A) = 1, whatever the base rate, and say what this looks like on the tree.
18. Show that if P(A \mid B) = P(A \mid \text{not } B), with 0 < P(B) < 1, then P(B \mid A) = P(B). Explain what this says about a check that flags both kinds of item at the same rate.
19. In Example 4, explain why the answer would be too high if the “second inspection” were just the same camera photographing the board again, so that a speck of dust that fooled it the first time fools it every time.