Running a simulation and estimating a probability
A simulation has been designed. How do its trials become a number, and how far can that number be trusted?
Where you are 14 in Basic probability
When you reach for it
Verena shoots rounds of three arrows at an archery club. How likely is she to put at least two arrows of a round in the gold? A simulation for this can be designed as in designing a simulation: a random digit for each arrow, three digits for a round. What is left is to run it, keep a record, and turn the record into a number.
That number is an estimate of a probability. Reach for one when counting the outcomes of a compound event is slow or out of reach: a round of arrows, a trip of several fish, a wait for a first success, a set of stickers to collect. It also checks an exact answer from a tree diagram, since the two should agree.
The procedure
Day 1: running trials and recording them
Run the design exactly as it was fixed. The assignment, the length of a trial and the starting place are settled before any digit is read. Read in order, cut the digits into trials, and judge a trial only once all of it is in. A trial cut off by the end of the digits is not counted, because nobody knows how it would have ended.
Record every trial. A results table has one line per trial: what the device gave and what that means. Record a count (the number of golds) rather than only yes or no where you can, since a count answers several questions later.
Gather the record in a tally table. List every possible result once and make one stroke \mathsf{I} for each trial that gave it, drawing every fifth stroke across the other four, \cancel{\mathsf{IIII}}. A result’s frequency is its number of strokes. Every trial lands in exactly one row, so the frequencies add up to the number of trials, which checks the tallying.
Divide. The estimate is the fraction of trials in which the event happened:
The sign \approx is read “is approximately equal to”. This is the relative frequency of the event. A fraction is the most honest way to write it: \displaystyle \frac{4}{10} shows that only ten trials stand behind it, while 0.4 hides that.
An estimate of 0 or 1 is still an estimate. If no trial gave the event, the estimate is 0, but the event need not be impossible: rare events often miss a short run.
Day 2: more trials, exact answers, and questions with no easy formula
Pooling runs. Groups running the same design on different digits add the successes and add the trials:
This is what one long run of all those trials would give. The mean of the groups’ estimates differs unless every group ran the same number of trials, because it gives a short run as much say as a long one.
Why more trials give a better estimate. With n trials, one trial going the other way moves the estimate by \displaystyle \frac{1}{n}: by 0.1 with 10 trials, by 0.01 with 100. So short runs scatter widely and long runs settle close together, as the law of large numbers describes.
Comparing with an exact answer. When a tree, a table or the multiplication rule gives the exact probability, multiply it by the number of trials: that is the count a perfect run would show. The gap from your count, measured in trials, says how far off the run was. A trial or two is ordinary luck. Dozens of trials in a run of hundreds is a reason to recheck the design and the exact calculation. Neither is a proof.
When simulating is easier than calculating. Waiting for a first success, a trial runs until the success comes, so its tree never ends. Collecting a set, the orders in which packets can arrive multiply very fast. A simulation handles both like any other question; only the end of a trial changes.
Worked examples
Day 1: three arrows on a row of digits
In the model for Verena’s archery, each arrow lands in the gold with probability 0.4, whatever the other arrows did. The design: one random digit per arrow, 0 to 3 gold and 4 to 9 not; one trial is three digits, one round; record the number of golds. Run ten trials on these digits and estimate the probability of at least two golds in a round.
Four digits of ten mean gold, matching 0.4. Read three digits at a time; the spaces are only a reading aid, so the second round, 9, 1, 6, crosses from one printed group into the next.
| Round | Digits | Golds | At least two? |
|---|---|---|---|
| 1 | 731 | 2 | yes |
| 2 | 916 | 1 | no |
| 3 | 751 | 1 | no |
| 4 | 712 | 2 | yes |
| 5 | 390 | 2 | yes |
| 6 | 031 | 3 | yes |
| 7 | 298 | 1 | no |
| 8 | 426 | 1 | no |
| 9 | 963 | 1 | no |
| 10 | 993 | 1 | no |
The event happened in 4 of the 10 trials, so
The count answers other questions too: every round had a gold, so this run’s estimate of “no gold at all” is \displaystyle \frac{0}{10} = 0.
Day 1: a tally table for a fishing trip
Ulric fishes a lake where, in the model, each fish he lands is a perch with probability 0.5, a bream with 0.3 and a pike with 0.2, independently of the others. A trip is four fish. The design: 0 to 4 perch (P), 5 to 7 bream (B), 8 and 9 pike (K); one trial is four digits; record the number of different species. Run twenty trips and estimate the probability of (a) all three species, (b) only one species, (c) at least two species.
Five, three and two digits of ten give 0.5, 0.3 and 0.2. Each printed row has twenty digits, exactly five trips. The first row reads:
| Trip | Digits | Fish | Species |
|---|---|---|---|
| 1 | 1079 | \mathrm{PPBK} | 3 |
| 2 | 3094 | \mathrm{PKPP} | 2 |
| 3 | 9303 | \mathrm{KPPP} | 2 |
| 4 | 5121 | \mathrm{BPPP} | 2 |
| 5 | 1859 | \mathrm{PKBK} | 3 |
Row by row, the twenty trips give 3, 2, 2, 2, 3; then 3, 2, 3, 2, 3; then 3, 2, 2, 1, 2; then 3, 2, 3, 2, 2 species. The one trip with a single species is 0211, four perch.
| Species | Tally | Frequency |
|---|---|---|
| 1 | \mathsf{I} | 1 |
| 2 | \cancel{\mathsf{IIII}}\;\cancel{\mathsf{IIII}}\;\mathsf{I} | 11 |
| 3 | \cancel{\mathsf{IIII}}\;\mathsf{III} | 8 |
| total | 20 |
The frequencies add to 20, so no trip was missed or counted twice.
(a) \displaystyle P(\text{all three species}) \approx \frac{8}{20} = 0.4.
(b) \displaystyle P(\text{only one species}) \approx \frac{1}{20} = 0.05.
(c) At least two is the second row and the third: \displaystyle \frac{11 + 8}{20} = \frac{19}{20} = 0.95. It is also 1 - 0.05, since “at least two species” is the complement of “only one”.
Day 1: a game of cups, run with cards
In a party game a ball is hidden under one of four cups, and Tiberia, with no clue where, points to one, so each round she is right with probability \displaystyle \frac{1}{4}, independently of other rounds. A game is three rounds, and she wins a prize if she is right at least once. She simulates with four cards, one with a ball drawn on it: shuffle, draw, write \mathrm{B} for the ball card (a right guess) or \mathrm{N} for any other, and put the card back. One trial is three draws. Her twelve trials:
Estimate the probability that she wins a prize.
SolutionPutting the card back keeps every draw at \displaystyle \frac{1}{4}; without that, the rounds would become dependent events, which the model says they are not. The right guesses per game are 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1.
| Right guesses | Tally | Frequency |
|---|---|---|
| 0 | \mathsf{IIII} | 4 |
| 1 | \cancel{\mathsf{IIII}}\;\mathsf{III} | 8 |
| 2 or 3 | 0 | |
| total | 12 |
A prize needs at least one right guess, which happened in 8 of the 12 games:
No game here had two right guesses, so this run puts that chance at 0, although such a game is plainly possible. Twelve trials are too few to show it.
Day 2: pooling five groups at the archery club
Verena’s ten rounds in Example 1 were one group’s run. Four more groups ran the same design on other digits, with these rounds showing at least two golds: Sorrel, 6 of 20; Tiberia, 9 of 20; Ulric, 7 of 25; Winthrop, 9 of 25. (a) Find each group’s estimate and the pooled estimate. (b) Find the exact probability in the model and compare.
Solution(a) Each estimate is a group’s successes over its own trials. To pool, add both columns, then divide once. The column “Yes” counts rounds with at least two golds.
| Group | Trials | Yes | Estimate |
|---|---|---|---|
| Verena | 10 | 4 | 0.40 |
| Sorrel | 20 | 6 | 0.30 |
| Tiberia | 20 | 9 | 0.45 |
| Ulric | 25 | 7 | 0.28 |
| Winthrop | 25 | 9 | 0.36 |
| all five | 100 | 35 | 0.35 |
The pooled estimate is \displaystyle \frac{35}{100} = 0.35. The five estimates add to 1.79, and their mean, \displaystyle \frac{1.79}{5} = 0.358, differs because it counts Verena’s ten rounds as heavily as Ulric’s twenty-five.
(b) A tree for one round has three stages, gold (0.4) or not (0.6) at each. Exactly two golds happens on three paths, GGN, GNG and NGG, each with probability 0.4 \times 0.4 \times 0.6 = 0.096 by the multiplication rule. All three gold, GGG, has 0.4^3 = 0.064. The paths are separate, so their probabilities add:
A perfect run of 100 would show 35.2 successes; the pooled run showed 35. The single groups ranged from 0.28 to 0.45, yet none did anything wrong. Tiberia, the furthest above, had 9 where a perfect run of 20 has 20 \times 0.352 = 7.04: about two trials.
Day 2: is the gap too big?
For the cups game of Example 3, find the exact probability of a prize. Then judge two classes’ runs of 400 games: Yvaine’s class got 230 prizes and Winthrop’s class 176. One class made a mistake. Which fits better: (i) each trial was only two draws, or (ii) the cards were not put back?
SolutionNo prize means three wrong guesses in a row, with probability \displaystyle \left(\frac{3}{4}\right)^3 = \frac{27}{64} by the multiplication rule. A prize is the complement:
A perfect run of 400 shows 231.25 prizes. Yvaine’s class is about one game off. Winthrop’s class is about 55 games short, where a run of the same size with the correct design missed by one: a reason to look for a mistake.
(i) With two draws, a prize has probability \displaystyle 1 - \left(\frac{3}{4}\right)^2 = \frac{7}{16}, so 400 games show about \displaystyle 400 \times \frac{7}{16} = 175.
(ii) Drawing three of the four cards without putting any back misses the ball card only when it is the one left over, so a prize has probability \displaystyle \frac{3}{4}, and 400 games show about 300.
The class got 176, so mistake (i) fits.
Day 2: waiting to leave the start
In Yvaine’s board game, a counter leaves the start square only when the spinner lands on go. The spinner has ten equal sections, three marked go, so each spin gives go with probability 0.3.
How likely is a player to need four or more spins to leave, and how many spins does it take on average? The design: 0, 1, 2 mean go; a trial reads digits until the first go; record the number of spins. Run it on:
A trial has no fixed length here. The first reads 3, 9, 5, 0: go on the fourth spin, so it records 4, and the next trial starts at the next digit. The trials are 3950, 2, 472, 886562, 1, 42, 2, 2, 72, 399670, 78630, 92, 540, 5859981 and 5860. The final 8, 8 start a trial the row does not finish, so it is not counted. The numbers of spins are 4, 1, 3, 6, 1, 2, 1, 1, 2, 6, 5, 2, 3, 7, 4, and six of them are four or more:
Exactly, four or more spins means missing go on each of the first three, with probability 0.7^3 = 0.343. A perfect run of 15 shows about 5.1; this run showed 6, about one trial more.
For the average, share the spins over the trials: 48 spins in 15 trials, so the mean wait is \displaystyle \frac{48}{15} = 3.2 spins. No rule met so far gives this, yet the same record answered it. (Looking ahead: the geometric distribution puts the exact mean at \displaystyle 3\frac{1}{3} spins.)
Day 2: collecting a set of four stickers
Each packet of a sticker series holds one of four animal stickers: fox, owl, hare or deer, each equally likely whatever came before. Sorrel buys packets one at a time until she has all four. Estimate the probability that she completes the set within eight packets. The design: 0 and 1 fox, 2 and 3 owl, 4 and 5 hare, 6 and 7 deer, 8 and 9 skipped; a trial reads digits until all four have appeared; record the number of packets.
Each sticker has two of the eight digits in use, so each has probability \displaystyle \frac{2}{8} = \frac{1}{4}. The first trial reads 4, 5, 3, 5, 5, skips the 8, then reads 5, 2, 5, 5, 7, 7, 0. The fox, 0, arrives with the twelfth packet and completes the set, so the trial records 12.
Continuing, the nine complete trials need 12, 5, 8, 8, 7, 16, 15, 6, 8 packets. The last four digits, 8, 1, 2, 0, begin a tenth trial that never finishes. Eight packets or fewer happened in 6 of the 9 trials:
An exact answer is harder. A tree for eight packets has 4^8 = 65{,}536 paths, and sorting out those that hold all four stickers takes methods met much later; they give about 0.62. The simulation needed none of that, though nine trials is a short run.
Boundaries
An estimate is not the probability. Another run gives another number. Report the number of trials with it, best as a fraction.
A simulation estimates the model’s probability. If Verena really hits the gold half the time, a design built on 0.4 estimates the wrong thing however many trials it has.
Trials must be separate and alike. Each trial uses fresh digits in order, built the same way. Runs may be pooled only if they used the same design for the same event.
Only finished trials count. In waiting and collecting problems, the end of almost every row cuts a trial short.
A count of 0 does not mean impossible. Three golds, with probability 0.064, is expected less than once in ten rounds. Rare events need long runs.
Judging a gap is a judgment. A wrong design can land near the exact answer by luck, and a correct one can, rarely, land far away.
Where this goes next
Waiting problems continue in simulating a waiting time, and their exact answers come from the geometric distribution. The scatter of the groups in Example 4 is the start of sampling distributions: repeat a run of 20 trials again and again, and the estimates spread out in a pattern of their own, which is how statisticians judge how far an estimate can be trusted.
Exercises
Day 1: running a design
- Use the archery design of Example 1 (0 to 3 gold, three digits a round) on 39801 \quad 76700 \quad 57985 33758 \quad 46339 \quad 25017. Record the number of golds in each of the ten rounds, and estimate the probability of three golds in a round.
- From the same ten rounds, estimate the probability of no gold, and of at least one gold.
- Use Ulric’s design from Example 2 on 47118 \quad 93837 \quad 88042 \quad 91286. Record the number of species on each trip, then estimate the probability of all three species, and of at least one pike.
- A board-game spinner has six equal sections: three marked 1, two marked 2 and one marked 3. It is lost, so Verena rolls a number cube instead: 1, 2 or 3 means move 1; 4 or 5 means move 2; 6 means move 3. A turn is two spins, and the event is “move at least 4 squares in the turn”. Her sixteen turns gave the rolls (5,2), (5,3), (3,3), (2,6), (3,6), (4,4), (1,5), (6,6), (4,5), (5,2), (5,5), (1,3), (3,5), (5,1), (4,1), (4,1). Find the move for each turn and estimate the probability.
Day 1: from a record to an estimate
- Winthrop’s class ran Ulric’s fishing design and tallied the species per trip: one species, frequency 2; two species, frequency 27; three species, frequency 21. How many trips were run? Estimate the probability of all three species as a fraction, a decimal and a percent, and the probability of more than one species.
- A simulation gives the event in 13 of 40 trials. Write the estimate as a fraction, a decimal and a percent. How many successes in 40 trials would have given an estimate of 0.35?
Day 2: pooling runs
- Four groups ran the fishing design for “at least one pike in a trip”, with 11 of 20, 13 of 20, 27 of 50 and 62 of 110 trips. Find the pooled estimate, and the mean of the four estimates to three decimal places. Which should be reported?
- One group ran 20 trials and another 200. By how much does one more success change each group’s estimate?
- Two groups ran the archery design for “at least two golds” and got 5 of 20 and 9 of 20. Pool them. Using the exact 0.352 of Example 4, how many successes would a perfect run of 20 show, and does either group look wrong?
Day 2: comparing with an exact answer
- In the archery model, find the exact probability of no gold in a round. A class pooled 200 rounds and 41 had no gold. Compare.
- For the board game of Exercise 4, list the 36 equally likely pairs of cube rolls and find the exact probability of moving at least 4 squares. Compare it with Verena’s 16 turns.
- In the archery model, find the exact probability of three golds in a round. Exercise 1 gave an estimate of 0 from 10 rounds. Is that a sign of a mistake? About how many rounds would a run need for about 10 of them to be expected to show three golds?
- Ulric’s exact probability of at least one pike in a trip of four fish is 1 - 0.8^4. A class pooled 300 trips and found a pike in 104. Compare, then decide which mistake fits the count: (a) only the digit 9 meant pike, or (b) each trip was three digits.
Day 2: waiting and collecting
- Run Yvaine’s go spinner from Example 6 on 64192 \quad 67457 \quad 66799 \quad 18146 15910 \quad 68784 \quad 46692 \quad 05904. Record the spins for each complete trial. Estimate the probability of needing four or more spins, the probability of leaving on the first spin, and the mean wait.
- A class pooled 100 trials of the go spinner, with these frequencies: 1 spin, 30; 2 spins, 22; 3 spins, 14; 4 spins, 11; 5 spins, 8; 6 or more spins, 15. Estimate the probability of needing four or more spins, and of leaving within two spins, and compare each with its exact value.
- Run Sorrel’s sticker design from Example 7 on 56532 \quad 22829 \quad 62341 \quad 68498 33404 \quad 08995 \quad 90633 \quad 10146 21897 \quad 39577 \quad 22007 \quad 45848. Record the packets needed in each complete trial and estimate the probability of needing more than eight.
- A smaller series has three stickers, equally likely. Simulate packets with 0, 1, 2 for the first sticker, 3, 4, 5 for the second, 6, 7, 8 for the third, and 9 skipped; a trial is three packets, and the event is “all three stickers are different”. Run as many trials as 78537 \quad 78760 \quad 51787 81459 \quad 80388 \quad 05388 allows and estimate the probability. Then find the exact probability by listing, and compare.
Reasoning
- Explain why runs are pooled by adding successes and adding trials, rather than by taking the mean of the estimates. When do the two agree?
- Two estimates are both 0.1 away from an exact answer, one from 10 trials and one from 1{,}000. Explain why only one of them is a reason to recheck the design.
- Explain why the sticker question of Example 7 is hard to answer with a tree, yet no harder to simulate than the archery of Example 1.
- Verena has run 30 trials from a row of random digits. To get 60, she reads the same row again from the start. Explain what this does to her estimate and why it does not help.
Answers
- The rounds 398, 017, 670, 057, 985, 337, 584, 633, 925, 017 have 1, 2, 1, 1, 0, 2, 0, 2, 1, 2 golds. None has three: \displaystyle \frac{0}{10} = 0.
- No gold: \displaystyle \frac{2}{10} = 0.2 (rounds 985 and 584). At least one: \displaystyle \frac{8}{10} = 0.8, the complement.
- The trips 4711, 8938, 3788, 0429, 1286 have 2, 2, 3, 2, 3 species. All three: \displaystyle \frac{2}{5} = 0.4. At least one pike: \displaystyle \frac{4}{5} = 0.8; only the first trip has no 8 or 9.
- The moves are 3, 3, 2, 4, 4, 4, 3, 6, 4, 3, 4, 2, 3, 3, 3, 3; for example (2,6) is 1 + 3 = 4. Six turns move at least 4: \displaystyle \frac{6}{16} = \frac{3}{8} = 0.375.
- 2 + 27 + 21 = 50 trips. All three species: \displaystyle \frac{21}{50} = 0.42 = 42\%. More than one: \displaystyle \frac{48}{50} = 0.96.
- \displaystyle \frac{13}{40} = 0.325 = 32.5\%. An estimate of 0.35 needs 0.35 \times 40 = 14 successes.
- Pooled: \displaystyle \frac{113}{200} = 0.565. The mean of 0.55, 0.65, 0.54 and \displaystyle \frac{62}{110} \approx 0.564 is about 0.576. Report 0.565; the mean gives each 20-trial group as much weight as the 110-trial group.
- By \displaystyle \frac{1}{20} = 0.05 for the group of 20, and by \displaystyle \frac{1}{200} = 0.005 for the group of 200.
- Pooled: \displaystyle \frac{14}{40} = 0.35. A perfect run of 20 shows 20 \times 0.352 = 7.04, so each group is about 2 trials away. Neither looks wrong.
- 0.6^3 = 0.216. The class estimate is \displaystyle \frac{41}{200} = 0.205. A perfect run shows 200 \times 0.216 = 43.2, so the class is about 2 trials short in 200: they agree.
- \displaystyle \frac{15}{36} = \frac{5}{12} \approx 0.417. Moves of at least 4 come from 1 + 3 or 3 + 1 (3 pairs each), 2 + 2 (4 pairs), 2 + 3 or 3 + 2 (2 pairs each) and 3 + 3 (1 pair): 3 + 3 + 4 + 2 + 2 + 1 = 15. A perfect run of 16 shows \displaystyle 16 \times \frac{5}{12} \approx 6.7; Verena got 6.
- 0.4^3 = 0.064. Ten rounds should show 10 \times 0.064 = 0.64 of them, less than one, so 0 is no sign of a mistake. About \displaystyle \frac{10}{0.064} = 156.25, so roughly 156 rounds.
- 1 - 0.8^4 = 0.5904, and a perfect run of 300 shows 300 \times 0.5904 = 177.12, so 104 is about 73 short: recheck. Mistake (a) gives 1 - 0.9^4 = 0.3439 and 300 \times 0.3439 = 103.17; mistake (b) gives 1 - 0.8^3 = 0.488 and about 146.4. So (a) fits.
- The waits are 3, 2, 11, 2, 3, 3, 1, 10, 1, 3 spins in ten complete trials; the final 4 starts an unfinished trial. Four or more: \displaystyle \frac{2}{10} = 0.2. First spin: \displaystyle \frac{2}{10} = 0.2. Mean: \displaystyle \frac{39}{10} = 3.9 spins.
- Four or more: \displaystyle \frac{34}{100} = 0.34, against the exact 0.7^3 = 0.343. Within two: \displaystyle \frac{52}{100} = 0.52, against the exact 0.3 + 0.7 \times 0.3 = 0.51. Both are within about one trial of a perfect run.
- The six complete trials need 13, 6, 6, 6, 5, 8 packets; the last two used digits start an unfinished trial. More than eight: \displaystyle \frac{1}{6} \approx 0.17.
- Nine trials, 785, 377, 876, 051, 787, 814, 580, 388, 053, with two digits left over. Only 814 and 580 have three different stickers: \displaystyle \frac{2}{9}. Exactly, of the 3 \times 3 \times 3 = 27 equally likely orders, 3 \times 2 \times 1 = 6 are all different: \displaystyle \frac{6}{27} = \frac{2}{9}. The exact match is luck.
- Adding successes and trials counts every trial once, giving what one long run of all of them would. The mean of the estimates gives a short run the same weight as a long one. The two agree when every run has the same number of trials.
- With 10 trials, a gap of 0.1 is one trial going the other way, which happens all the time. With 1{,}000 trials it is 100 trials going the same way, far more than chance usually moves. That calls for a recheck, though it proves nothing.
- A tree needs a stage for every packet: for eight packets it has 4^8 = 65{,}536 paths, and for “until the set is complete” it has no last stage. A simulation never builds the tree. Each trial is one path, read off the digits and counted as for the archery, so the work grows with the number of trials, not with how complicated the event is.
- Reading the same digits again gives the same 30 trials with the same results. Successes and trials both double, so the estimate does not change, and it still rests on only 30 chance results. New trials need new digits.