Conditioning and Bayes' rule, rigorously
When we learn that an event has happened, how should every other probability change, and how do we turn the chance of the evidence under each explanation into the chance of each explanation given the evidence?
Where you are 4 in Foundations
Why this exists
The school lessons Finding a conditional probability and Bayes' rule find conditional probabilities from tables and trees. A course built on the axioms of Probability axioms and sample spaces needs more than the computation. Once we condition on an event, do the complement rule, the addition rule and every other result still hold for the new probabilities? What replaces the tree when the situations that can produce the evidence are infinitely many? Does updating on two pieces of evidence one after the other give the same answer as updating on both at once? And when two descriptions of what was learned sound alike (“at least one is short-dated”, “the first one is short-dated”), which event is being conditioned on?
This lesson answers all four from the axioms. The answer to the first is the key: conditioning on B produces a new probability, so everything already proved applies to it.
Definition
Fix a probability space (\Omega,P) and an event B with P(B)>0. The conditional probability of A given B is
This is a definition, not a theorem; the Intuition and Result 1 show why it is the right one. When P(B)=0 the quotient is undefined (see Boundaries).
A partition of \Omega is a finite or countably infinite collection of events B_1,B_2,\dots that are pairwise disjoint and whose union is \Omega: every outcome lies in exactly one B_i.
In Bayes’ rule the B_i are called hypotheses and the observed event A the evidence. P(B_i) is the prior probability of B_i, P(A\mid B_i) the likelihood of the evidence under B_i, and P(B_i\mid A) the posterior probability of B_i.
For an event H with P(H)<1, the odds of H are O(H)=P(H)/P(H^c), as in Odds, and given E with P(E\cap H^c)>0 they are O(H\mid E)=P(H\mid E)/P(H^c\mid E). Odds o convert back to the probability o/(1+o). The likelihood ratio of E for H is P(E\mid H)/P(E\mid H^c), when P(E\mid H^c)>0.
Intuition
Learning that B occurred rules out every outcome outside B. The probability mass that remains, on B, totals only P(B), so it is scaled up by the factor 1/P(B) to total 1 again. Nothing else changes: an event inside B that was twice as likely as another is still twice as likely. On a discrete \Omega this reads P(\{\omega\}\mid B)=p(\omega)/P(B) for \omega\in B and 0 otherwise. Figure 1 shows it for the distribution of Example 1.
Long-run frequencies give the same formula. In n repetitions of an experiment, the fraction of the repetitions in which B occurred that also had A is n_{A\cap B}/n_B=f_n(A\cap B)/f_n(B), the ratio of two relative frequencies.
Results
Throughout, (\Omega,P) is a probability space.
Result 1 (conditioning gives a probability). If P(B)>0, then Q(A)=P(A\mid B) is a probability on \Omega, with Q(B)=1. It is the only probability Q with Q(B)=1 for which there is a constant c with Q(A)=cP(A) for every event A\subseteq B. So every result of Probability axioms and sample spaces holds given B; for example P(A^c\mid B)=1-P(A\mid B).
Result 2 (multiplication rule). If P(A_1\cap\dots\cap A_{n-1})>0, then
The case n=2, P(A\cap B)=P(B)P(A\mid B), is The general multiplication rule.
Result 3 (The law of total probability). If B_1,B_2,\dots is a finite or countable partition with every P(B_i)>0, then for every event A
Result 4 (Bayes’ rule). Under the hypotheses of Result 3, if P(A)>0 then for each j
Result 5 (odds form). If 0<P(H)<1, P(E)>0 and P(E\mid H^c)>0, then
Posterior odds are prior odds times the likelihood ratio.
Result 6 (conditioning in stages). If P(B\cap C)>0 and Q=P(\cdot\mid B), then Q(C)>0 and Q(A\mid C)=P(A\mid B\cap C) for every event A. Updating on B and then on C gives the same probabilities as updating on B\cap C at once.
Why it is true
Result 1. Q(A)\ge 0 because it is a quotient of a nonnegative number by a positive one, and Q(\Omega)=P(B)/P(B)=1. If A_1,A_2,\dots are pairwise disjoint, so are A_1\cap B,A_2\cap B,\dots, and the distributive law gives \left(\bigcup_i A_i\right)\cap B=\bigcup_i(A_i\cap B). Countable additivity of P, divided by P(B), gives Q\left(\bigcup_i A_i\right)=\sum_i Q(A_i). Finally Q(B)=P(B)/P(B)=1.
For uniqueness, let Q be a probability with Q(B)=1 and Q(A)=cP(A) for A\subseteq B. Taking A=B gives c=1/P(B). For any event A, monotonicity gives Q(A\cap B^c)\le Q(B^c)=0, so additivity gives Q(A)=Q(A\cap B), which is P(A\cap B)/P(B).
Result 2. For k\le n-1, the event A_1\cap\dots\cap A_k contains A_1\cap\dots\cap A_{n-1}, so by monotonicity it has positive probability and every conditional probability in the product is defined. Write C_k=A_1\cap\dots\cap A_k. By the definition each factor after the first is P(A_k\mid C_{k-1})=P(C_k)/P(C_{k-1}), so the product telescopes:
Result 3. Every outcome of A lies in exactly one B_i, so A is the union of the pairwise disjoint events A\cap B_i. Countable additivity (finite additivity for a finite partition) gives P(A)=\sum_i P(A\cap B_i), and P(A\cap B_i)=P(A\mid B_i)P(B_i) by the definition.
Result 4. By the definition, P(B_j\mid A)=P(A\cap B_j)/P(A). Write the numerator as P(A\mid B_j)P(B_j) and the denominator by Result 3.
Result 5. H and H^c partition \Omega and both have positive probability. Result 4 with this partition gives P(H\mid E) and P(H^c\mid E) over the same denominator P(E). Dividing the first by the second cancels it:
The denominator on the left is positive, because P(H^c\mid E) is P(E\mid H^c)P(H^c)/P(E), a quotient of positive numbers.
Result 6. Q(C)=P(B\cap C)/P(B)>0. Then
Worked examples
Conditioning a distribution
The number N of prescriptions in a customer’s order at Priya’s pharmacy has P(N=1)=0.40, P(N=2)=0.25, P(N=3)=0.15, P(N=4)=0.10, P(N=5)=0.06 and P(N=6)=0.04. Find the probabilities of the values of N given N\ge 3, then P(N\ge 5\mid N\ge 3), P(N\le 4\mid N\ge 3) and P(N\text{ even}\mid N\ge 3).
SolutionLet B=\{N\ge 3\}, so P(B)=0.15+0.10+0.06+0.04=0.35. For n\ge 3 the event \{N=n\} lies inside B, so P(N=n\mid B)=P(N=n)/0.35:
For n=1,2 the intersection with B is empty and the conditional probability is 0. The four values total \tfrac{15+10+6+4}{35}=1, as Result 1 requires (Figure 1).
Adding values, P(N\ge 5\mid B)=\tfrac{6+4}{35}=\tfrac{2}{7}. The complement rule holds given B by Result 1, so P(N\le 4\mid B)=1-\tfrac{2}{7}=\tfrac{5}{7}. And P(N\text{ even}\mid B)=\tfrac{10+4}{35}=\tfrac{2}{5}, against P(N\text{ even})=0.25+0.10+0.04=0.39 without the condition.
What exactly was learned
Priya’s shelf holds 10 bottles of an antibiotic, 3 of them short-dated. Omar takes 3 bottles at random, one after another. Find the probability that at least two are short-dated given (a) at least one of the three is short-dated, and (b) the first bottle he takes is short-dated.
SolutionTake \Omega to be the ordered triples of distinct bottles, 10\cdot 9\cdot 8=720 of them, equally likely, as in Counting models. In a uniform model the definition becomes a ratio of counts, P(A\mid B)=|A\cap B|/|B|, since the factors 1/|\Omega| cancel.
Let T be “at least two short-dated”, L “at least one” and F “the first is short-dated”. Triples with no short-dated bottle number 7\cdot 6\cdot 5=210, so |L|=510. Exactly two short-dated: 3 places for the other bottle, then 3\cdot 2\cdot 7 ordered choices, giving 126; all three: 3\cdot 2\cdot 1=6. So |T|=132, and since T\subseteq L,
For (b), |F|=3\cdot 9\cdot 8=216, and the triples in F with no further short-dated bottle number 3\cdot 7\cdot 6=126, so |T\cap F|=90 and
Both answers are correct; they condition on different events. F is a smaller event inside L, and it says more: a named bottle is short-dated, which leaves two short-dated bottles among the other nine.
A product of conditional probabilities
A carton of 15 vials holds 2 cracked ones, and Omar inspects the vials one at a time in a random order, every order equally likely. Find the probability that the first cracked vial is the fourth one inspected, and the probability that the first 10 vials are all intact.
SolutionLet G_i be “the ith vial is intact”. Orders whose first k vials are intact number 13\cdot 12\cdots(14-k)\cdot(15-k)!, so
In words: after k intact vials, the next is equally likely to be any of the 15-k left, and 2 of them are cracked. The first cracked vial is the fourth exactly on W=G_1\cap G_2\cap G_3\cap G_4^c, and Result 2 gives
For the second question the product telescopes:
Check: the positions of the two cracked vials are a random pair from \binom{15}{2}=105. The first event needs one at 4 and the other among positions 5 to 15, 11 pairs; the second needs both among positions 11 to 15, \binom{5}{2}=10 pairs. So \tfrac{11}{105} and \tfrac{10}{105}=\tfrac{2}{21}.
Infinitely many hypotheses
During a quiet 10-minute stretch, the number K of customers who come to Priya’s counter has P(K=k)=\left(\tfrac{1}{2}\right)^{k+1} for k=0,1,2,\dots. Given K=k, each customer, independently of the others, needs a consultation with probability \tfrac{1}{3}. Find the probability P(A) that nobody needs a consultation, and the distribution of K given A.
SolutionThe values \left(\tfrac{1}{2}\right)^{k+1} are positive and sum to 1 by the geometric series, so the events \{K=k\} form a countable partition of positive-probability parts. Given K=k, the event A needs all k customers to go without a consultation, so by Independent events and the multiplication rule P(A\mid K=k)=\left(\tfrac{2}{3}\right)^k; for k=0 this is 1, correctly. No tree can show all the cases, but Result 3 needs no tree:
Countable additivity is what allows a sum of infinitely many terms here. Bayes’ rule, Result 4, gives each posterior:
These total \tfrac{2}{3}\cdot\tfrac{3}{2}=1, as Result 1 requires. A quiet stretch shifts weight toward few customers: P(K=0\mid A)=\tfrac{2}{3} against P(K=0)=\tfrac{1}{2}, and P(K\ge 2\mid A)=1-\tfrac{2}{3}-\tfrac{2}{9}=\tfrac{1}{9} against P(K\ge 2)=\tfrac{1}{4}.
Repeated evidence in odds
Priya receives an expensive injectable medicine from a wholesaler she has not used before. Suppose 1 package in 200 from such a source is counterfeit. A field test is positive on 90\% of counterfeit packages and on 5\% of genuine ones, and tests on separate samples from one package are independent given that the package is counterfeit, and also given that it is genuine. Find the probability that a package is counterfeit after one positive test, after two, and how many positives raise it above 0.95.
SolutionLet H be “counterfeit” and E_i “the ith test is positive”. The independence assumed is conditional independence: the events E_i are independent for the probability P(\cdot\mid H) and also for P(\cdot\mid H^c), so that, for instance, P(E_1\cap E_2\mid H)=P(E_1\mid H)\,P(E_2\mid H). It is an assumption about how the tests behave, not a consequence of anything above, and Independence of events and variables treats it in full. The prior odds are O(H)=\tfrac{1/200}{199/200}=\tfrac{1}{199} and the likelihood ratio is \tfrac{0.90}{0.05}=18. By Result 5,
For the second test, let Q=P(\cdot\mid E_1), a probability by Result 1. By Result 6 and the independence assumption,
and in the same way Q(E_2\mid H^c)=0.05. Result 5 applied to Q, whose prior odds are \tfrac{18}{199}, and Result 6 give
After n positives the odds are 18^n/199. A probability above 0.95 means odds above 19, so 18^n>19\cdot 199=3781. Since 18^2=324 and 18^3=5832, three positives are needed, giving \tfrac{5832}{6031}\approx 0.967. A test that is right 90\% of the time on counterfeits leaves the probability near 0.08 after one positive, because the prior is so small.
Association runs both ways
Let A and B be events with P(A)>0 and P(B)>0. Prove that P(A\mid B)>P(A) exactly when P(B\mid A)>P(B), and that both ratios P(A\mid B)/P(A) and P(B\mid A)/P(B) are equal. Illustrate with P(A)=0.1, P(B)=0.3, P(A\cap B)=0.06, where A is “a customer buys a cold remedy” and B “a customer buys tissues”.
SolutionBy the definition,
The two ratios are one number, so one exceeds 1 exactly when the other does, which is the first claim.
With the figures given, P(A\mid B)=\tfrac{0.06}{0.3}=0.2, twice P(A), and P(B\mid A)=\tfrac{0.06}{0.1}=0.6, twice P(B). Tissue buyers are twice as likely as the average customer to buy a cold remedy, and cold remedy buyers twice as likely to buy tissues, though P(A\mid B)=0.2 and P(B\mid A)=0.6 differ.
Boundaries
- Events of probability zero. P(A\mid B) is undefined when P(B)=0. For a continuous random variable Y every event \{Y=y\} has probability 0, yet “given Y=y” is a natural condition; it is handled with densities in Joint distributions, and the answer depends on more than the event \{Y=y\} alone.
- The whole of what was learned. The event conditioned on must be everything the observer learned, including how it was learned. Example 2 shows two reports that sound alike giving \tfrac{22}{85} and \tfrac{5}{12}. When the report comes from a process (someone chooses what to reveal), that process must be built into \Omega first.
- Reversal needs the prior. P(A\mid B) and P(B\mid A) are equal only when P(A)=P(B) or P(A\cap B)=0 (Exercise 19). Bayes’ rule turns one into the other only with the priors P(B_i), and only for hypotheses that are disjoint and cover \Omega; leaving one out shrinks the denominator.
- Odds form. Result 5 needs 0<P(H)<1 and P(E\mid H^c)>0. If P(E\mid H^c)=0, the evidence rules out H^c and P(H\mid E)=1.
- Repeated evidence. Result 6 holds always, but likelihood ratios multiply as in Example 5 only when the pieces of evidence are independent given H and given H^c (conditional independence, taken up in Independence of events and variables).
Where this goes next
When P(A\mid B)=P(A), learning B changes nothing about A; that is independence, made precise for events and random variables in Independence of events and variables. Conditioning a probability mass function on an event, as in Example 1, is a step in Discrete distributions: binomial, geometric, Poisson, and conditioning one random variable on another gives the conditional distributions of Joint distributions. Averaging conditional answers over a partition, as in Result 3, returns as the tower rule of Conditional expectation.
Exercises
Conditioning on an event
- The number D of days until a special order reaches Priya’s pharmacy has P(D=d)=\tfrac{d}{15} for d=1,2,3,4,5. Find the probabilities of the values of D given D\ge 3, and P(D\le 4\mid D\ge 3).
- Let A be “a customer buys a cold remedy” and B “a customer picks up a prescription”, with P(A)=0.5, P(B)=0.4 and P(A\cup B)=0.75. Find P(A\mid B), P(B\mid A), P(A^c\mid B) and P(A\mid B^c).
- A bin holds 8 sample packs, 3 of a new flavor. Omar takes 2 at random, one after the other. Find the probability that both are the new flavor given that at least one is, and given that the first is.
- Given P(A\mid C)=0.6, P(B\mid C)=0.5 and P(A\cap B\mid C)=0.3, find P(A\cup B\mid C), P(A^c\cap B^c\mid C) and P(A\cap B^c\mid C).
Chains and partitions
- A roll of 12 labels contains 3 misprinted ones, in a random order with every order equally likely. Using Result 2, find the probability that the first misprinted label is the third one, and that the first 4 labels are all correct.
- Of the prescriptions Priya receives, 70\% arrive electronically and 30\% on paper. A prescription needs a call to the prescriber with probability 0.02 if electronic and 0.10 if paper. Find the probability that a prescription needs a call, and the probability that it was on paper given that it needed one.
- The number K of items in an order has P(K=k)=\tfrac{1}{3}\left(\tfrac{2}{3}\right)^{k-1} for k=1,2,3,\dots. Given K=k, each item independently needs a pharmacist’s signature with probability \tfrac{1}{4}. Let A be “no item needs a signature”. Find P(A) and P(K=k\mid A) for every k.
- Three suppliers provide 50\%, 30\% and 20\% of an item, and the first two ship a damaged box with probability 0.02 and 0.05. The overall probability of a damaged box is 0.04. Find the third supplier’s damage rate, and the probability that a damaged box came from the second supplier.
Bayes’ rule and odds
- Priya’s software flags a prescription as possibly forged. Suppose P(H)=0.002 for H “forged”, the software flags 96\% of forged prescriptions and 3\% of genuine ones. Find the likelihood ratio, the posterior odds and P(H\mid\text{flag}).
- A flagged prescription in Exercise 9 goes to a second check, which flags 90\% of forged prescriptions and 1\% of genuine ones, independently of the first check given H and given H^c. Find the probability that it is forged if both checks flag it.
- In Exercise 9, find the probability that a prescription is forged given that the software does not flag it.
- With the software of Exercise 9, what prior P(H) would make P(H\mid\text{flag})=\tfrac{1}{2}?
Proofs
- Let C be an event and B_1,B_2,\dots a partition with every P(B_i\cap C)>0. Prove that P(A\mid C) equals \sum_i P(A\mid B_i\cap C)\,P(B_i\mid C).
- Let B_1,B_2,\dots be a partition with every P(B_i)>0. Prove that if P(A\mid B_i)\ge c for every i, then P(A)\ge c, and that if P(A\mid B_i)=c for every i, then P(A)=c.
- Prove that if A\subseteq B and P(B)>0, then P(A\mid B)\ge P(A), with equality exactly when P(A)=0 or P(B)=1.
- Prove that if P(B)>0 then P(A\mid A\cup B)\ge P(A\mid B).
- Let 0<P(H)<1 and P(E)>0. Prove that P(H\mid E)>P(H) exactly when P(E\mid H)>P(E\mid H^c).
- Refute with a counterexample: “for any events A and B with 0<P(B)<1, P(A\mid B)+P(A\mid B^c)=1.” State the true identity that resembles it.
- Let P(A)>0 and P(B)>0. Prove that P(A\mid B)=P(B\mid A) exactly when P(A)=P(B) or P(A\cap B)=0.
Answers
- P(D\ge 3)=\tfrac{12}{15}, so given D\ge 3 the values 3,4,5 have \tfrac{3}{12}=\tfrac{1}{4}, \tfrac{4}{12}=\tfrac{1}{3}, \tfrac{5}{12}; P(D\le 4\mid D\ge 3)=\tfrac{7}{12}.
- P(A\cap B)=0.5+0.4-0.75=0.15. P(A\mid B)=\tfrac{3}{8}, P(B\mid A)=\tfrac{3}{10}, P(A^c\mid B)=1-\tfrac{3}{8}=\tfrac{5}{8} by Result 1, and P(A\mid B^c)=\tfrac{0.5-0.15}{0.6}=\tfrac{7}{12}.
- Of the 56 ordered pairs, 6 are both new, 56-20=36 have at least one new, and 3\cdot 7=21 start with a new one. So \tfrac{6}{36}=\tfrac{1}{6} and \tfrac{6}{21}=\tfrac{2}{7}.
- By Result 1 the addition and complement rules hold given C: P(A\cup B\mid C)=0.6+0.5-0.3=0.8, P(A^c\cap B^c\mid C)=1-0.8=0.2, P(A\cap B^c\mid C)=0.6-0.3=0.3.
- \tfrac{9}{12}\cdot\tfrac{8}{11}\cdot\tfrac{3}{10}=\tfrac{9}{55} and \tfrac{9}{12}\cdot\tfrac{8}{11}\cdot\tfrac{7}{10}\cdot\tfrac{6}{9}=\tfrac{14}{55}. Check: \tfrac{\binom{9}{2}}{\binom{12}{3}}=\tfrac{36}{220} and \tfrac{\binom{9}{4}}{\binom{12}{4}}=\tfrac{126}{495}.
- By Result 3, P(\text{call}) is 0.7\cdot 0.02+0.3\cdot 0.10=0.044; P(\text{paper}\mid\text{call})=\tfrac{0.030}{0.044}=\tfrac{15}{22} by Result 4.
- P(A\mid K=k)=\left(\tfrac{3}{4}\right)^k. The term \tfrac{1}{3}\left(\tfrac{2}{3}\right)^{k-1}\left(\tfrac{3}{4}\right)^k equals \tfrac{1}{4}\left(\tfrac{1}{2}\right)^{k-1}, so Result 3 and the geometric series give P(A)=\tfrac{1}{4}\cdot 2=\tfrac{1}{2}. By Result 4, P(K=k\mid A) is \tfrac{1}{4}\left(\tfrac{1}{2}\right)^{k-1} divided by \tfrac{1}{2}, which is \left(\tfrac{1}{2}\right)^k.
- 0.5\cdot 0.02+0.3\cdot 0.05+0.2x=0.04 gives x=0.075. Then P(\text{second}\mid\text{damaged})=\tfrac{0.015}{0.04}=\tfrac{3}{8}.
- Likelihood ratio \tfrac{0.96}{0.03}=32; prior odds \tfrac{0.002}{0.998}=\tfrac{1}{499}; posterior odds \tfrac{32}{499}; P(H\mid\text{flag})=\tfrac{32}{531}\approx 0.060.
- As in Example 5, the second check acts on the posterior odds with likelihood ratio \tfrac{0.90}{0.01}=90: odds \tfrac{32\cdot 90}{499}=\tfrac{2880}{499}, so the probability is \tfrac{2880}{3379}\approx 0.852.
- The likelihood ratio of “no flag” is \tfrac{0.04}{0.97}=\tfrac{4}{97}, so the odds are \tfrac{1}{499}\cdot\tfrac{4}{97}=\tfrac{4}{48{,}403} and the probability is \tfrac{4}{48{,}407}\approx 0.000083.
- Posterior odds 1 need prior odds \tfrac{1}{32}, so P(H)=\tfrac{1}{33}.
- By monotonicity P(C)\ge P(B_1\cap C)>0, so Q=P(\cdot\mid C) is defined and is a probability by Result 1, with Q(B_i)=P(B_i\mid C)>0. Result 3 for Q gives Q(A)=\sum_i Q(A\mid B_i)Q(B_i), and Q(A\mid B_i)=P(A\mid B_i\cap C) by Result 6.
- By Result 3 and the bound on each P(A\mid B_i), P(A)\ge c\sum_i P(B_i). The B_i are disjoint with union \Omega, so \sum_i P(B_i)=P(\Omega)=1 by countable additivity, and P(A)\ge c. If every P(A\mid B_i)=c, the same sum equals c\sum_i P(B_i)=c.
- A\cap B=A, so P(A\mid B)=P(A)/P(B). Since 0<P(B)\le 1, this is at least P(A), and it equals P(A) exactly when P(A)\left(1-P(B)\right)=0, that is, P(A)=0 or P(B)=1.
- Let x=P(A\cap B^c), y=P(A\cap B), z=P(A^c\cap B), all nonnegative, with y+z=P(B)>0. Then P(A\mid A\cup B)=\tfrac{x+y}{x+y+z} and P(A\mid B)=\tfrac{y}{y+z}. Both denominators are positive, and (x+y)(y+z) exceeds y(x+y+z) by xz\ge 0, which is the inequality after cross-multiplying.
- P(H\mid E)>P(H) means P(E\mid H)P(H)>P(E)P(H), that is, P(E\mid H)>P(E), since P(H)>0. Write q=P(H^c)>0. By Result 3, P(E) is P(E\mid H)(1-q) plus P(E\mid H^c)\,q, so the condition becomes q\,P(E\mid H)>q\,P(E\mid H^c), and dividing by q gives P(E\mid H)>P(E\mid H^c). Every step reverses.
- Take A=\Omega: the two terms are 1 and 1, totaling 2, for any B with 0<P(B)<1. The true identity is P(A\mid B)+P(A^c\mid B)=1, the complement rule given B (Result 1).
- Multiply \tfrac{P(A\cap B)}{P(B)}=\tfrac{P(A\cap B)}{P(A)} by P(A)P(B)>0: it is equivalent to P(A\cap B)\left(P(A)-P(B)\right)=0, which holds exactly when P(A\cap B)=0 or P(A)=P(B).