Sampling variability and simulating it
Two fair random samples from the same bag give different answers. Which one is right, and how far from the truth could either of them be?
Where you are 7 in Sampling distributions
Why this exists
A sample is taken because checking the whole population is too slow, too costly or impossible. The number the sample gives, a statistic, then stands in for the population’s number, the parameter.
But a second random sample of the same size gives a different statistic, and a third gives a third. None of them was done wrongly: chance decides which members land in each sample. So every estimate has to answer a question: if the next sample would have said something else, how much is this one worth?
Sampling variability is the name for that change from sample to sample. Drawing many samples from a population whose answer is known shows that it is not chaos: the statistics pile up around the true value, tighter when the samples are bigger. That pattern lets one sample say “the answer is probably within about this much of what I found”.
Definition
Sampling variability is the variation of a statistic from one random sample to another, when every sample has the same size and is drawn from the same population by the same method.
To see it, you simulate: draw many random samples of one size (with real objects, with random digits or a device that stands in for the real draw, or with a computer), work out the statistic for each, and plot all the values, usually as a dot plot with one dot per sample. Two features of that plot matter:
- its center, where the pile sits, read as its middle or as the mean of all the plotted values;
- its spread, how far the dots stretch on either side, described by the smallest and largest values or by how many dots land within some distance of the center.
“Within 0.1 of 0.6” means at least 0.5 and at most 0.7, both ends included, throughout this lesson.
The statistic can be a sample proportion, the share of the sample with some feature, or a sample mean, the mean of a measurement over the sample. (Later courses write these as \hat{p} and \bar{x}.)
Intuition
A bag holds 200 beads, and 120 of them are blue. The share of blue in the whole bag is the parameter:
Picture Damaris shaking the bag, taking out 10 beads without looking, counting the blue ones and putting all ten back, thirty times over. Each handful is a random sample of 10 and gives a sample proportion: 7 blue beads is 0.7, 4 blue beads is 0.4. Figure 1 shows one set of thirty such handfuls, simulated by computer, with one dot per handful.
The values differ: six different proportions came out of one bag that never changed. They gather around the bag’s share and thin out toward the edges (another run would shift the column heights a little). So one handful of 10 could easily be off by 0.1, and now and then by 0.2 or 0.3.
Results
1. Statistics vary. Different random samples of the same size from the same population usually give different values of a statistic. No sample is wrong because of this; it is what choosing by chance does.
2. Random samples center on the parameter. The dot plot of the statistics from many random samples piles up around the parameter. Most samples land near it, and they miss high about as often as they miss low.
3. Bigger samples spread less. With the population and the method kept the same, statistics from larger samples crowd closer to the parameter than statistics from smaller samples.
4. The spread says how close one estimate is likely to be. If most simulated samples of some size land within a distance d of the parameter, then one real sample of that size is likely to be within about d of the parameter too.
Why it is true
Why statistics vary. A sample holds only some of the population, and chance decides which members. Two handfuls can hold different mixtures of colors, so they give different proportions.
Why they center on the parameter. In a random sample every bead has the same chance of being drawn, and blue beads make up 0.6 of the bag, so nothing pushes the samples high or low. Listing every possible sample from a tiny population, as in Populations, samples and census, shows the exact version: the average of the statistic over all the possible samples equals the parameter. A simulation draws only some of them, so its center lands near the parameter rather than on it, as the 0.597 of Example 1 does.
Why bigger samples spread less. One bead changes the proportion of a sample of 10 by \displaystyle \frac{1}{10} = 0.1, but of a sample of 40 by only \displaystyle \frac{1}{40} = 0.025. To land 0.2 away from 0.6, a sample of 10 needs just 2 more blue beads than usual (or 2 fewer); a sample of 40 needs 8, all pushing the same way. Chance rarely lines up that many, because inside a big sample the draws that push up and the draws that push down tend to cancel. It is the same averaging that drives the law of large numbers.
Why the spread measures closeness. “The sample is within 0.1 of the truth” and “the truth is within 0.1 of the sample” say the same thing. So if most samples land within 0.1 of the truth, the truth is within 0.1 of most samples. Your own sample may be one that missed by more, so the claim is “likely”, never “certain”.
Worked examples
Day 1: reading a dot plot of thirty samples
Figure 1 shows thirty samples of 10 beads from the bag. Read off how many samples gave each proportion. Then find the center of the plot, its smallest and largest values, and how many samples landed within 0.1 of 0.6.
SolutionCount the dots in each column:
| Sample proportion | Samples |
|---|---|
| 0.3 | 2 |
| 0.4 | 4 |
| 0.5 | 3 |
| 0.6 | 9 |
| 0.7 | 8 |
| 0.8 | 4 |
The counts add to 2 + 4 + 3 + 9 + 8 + 4 = 30, one for each sample, as they must.
Center. The tallest column is at 0.6. For the mean of all thirty values, add each value as many times as it occurs:
Dividing by the 30 samples gives \displaystyle \frac{17.9}{30} \approx 0.597, close to the bag’s share, 0.6.
Spread. The values run from 0.3 to 0.8. Within 0.1 of 0.6 means 0.5, 0.6 or 0.7, which is 3 + 9 + 8 = 20 of the 30 samples. The other 10 missed by 0.2 or 0.3.
In this run, two samples in three landed within 0.1 of the truth. That describes these thirty samples only; a fresh run would give a somewhat different count.
Day 1: sample means of leaf lengths from random digits
Aldric measured all 100 leaves on a young tree to the nearest centimeter and numbered them 00 to 99. In the tables the row is a leaf’s first digit and the column its second digit, so leaf 37 is in row 3, column 7 of the second table, and is 5 cm long. The 100 lengths add to 800 cm.
| Row | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| 0 | 9 | 6 | 8 | 7 | 8 |
| 1 | 5 | 11 | 9 | 6 | 7 |
| 2 | 9 | 6 | 9 | 10 | 9 |
| 3 | 7 | 10 | 7 | 11 | 9 |
| 4 | 7 | 10 | 8 | 8 | 5 |
| 5 | 8 | 10 | 7 | 6 | 9 |
| 6 | 10 | 7 | 5 | 7 | 7 |
| 7 | 11 | 7 | 6 | 8 | 8 |
| 8 | 6 | 10 | 9 | 8 | 8 |
| 9 | 10 | 7 | 10 | 7 | 9 |
| Row | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|
| 0 | 11 | 8 | 9 | 8 | 11 |
| 1 | 9 | 8 | 11 | 7 | 10 |
| 2 | 6 | 7 | 7 | 7 | 5 |
| 3 | 7 | 7 | 5 | 5 | 7 |
| 4 | 8 | 8 | 8 | 8 | 9 |
| 5 | 10 | 10 | 7 | 7 | 5 |
| 6 | 7 | 11 | 7 | 6 | 9 |
| 7 | 8 | 6 | 7 | 10 | 5 |
| 8 | 10 | 11 | 8 | 8 | 8 |
| 9 | 10 | 9 | 6 | 8 | 10 |
He draws two random samples of 5 leaves, one from each row of random digits below. He reads each row in pairs, and skips a pair if that leaf is already in the sample.
Find each sample mean and compare it with the mean length of all 100 leaves.
SolutionThe parameter is the mean of the whole tree: \displaystyle \frac{800}{100} = 8 cm.
Each leaf has exactly one pair from 00 to 99 and every pair is equally likely, so this draws a simple random sample; skipping a repeat keeps a leaf from being measured twice.
The first row gives the pairs 64, 62, 88, 97, 51, with lengths 7, 5, 8, 6 and 10 cm, so the sample mean is
The second row gives 04, 43, 19, 82, 96, with lengths 8, 8, 10, 9 and 9 cm:
The first sample is 0.8 cm below the tree’s mean and the second 0.8 cm above it. A sample mean varies from sample to sample just as a sample proportion does.
Day 2: samples of 10 against samples of 40
Figure 2 puts thirty random samples of 40 beads, from the same bag, under the samples of 10 from Figure 1. Compare the two plots: their centers, their smallest and largest values, and how many samples landed within 0.1 of 0.6.
Reading the lower plot, with each proportion worked out from its count of blue beads:
| Blue beads | Sample proportion | Samples |
|---|---|---|
| 18 | 0.45 | 1 |
| 20 | 0.5 | 2 |
| 21 | 0.525 | 4 |
| 23 | 0.575 | 4 |
| 24 | 0.6 | 4 |
| 25 | 0.625 | 6 |
| 26 | 0.65 | 3 |
| 27 | 0.675 | 4 |
| 28 | 0.7 | 1 |
| 29 | 0.725 | 1 |
Centers. The thirty samples of 40 held 723 blue beads among 30 \times 40 = 1200 beads drawn. Since every sample has the same size, the mean of the thirty proportions is \displaystyle \frac{723}{1200} = 0.6025, against about 0.597 for the samples of 10. Both plots center near the bag’s share; making the samples bigger did not move the center.
Spread. The samples of 10 run from 0.3 to 0.8, a stretch of 0.5. The samples of 40 run from 0.45 to 0.725, a stretch of 0.275, a little more than half as wide. Within 0.1 of 0.6 lie 20 of the 30 samples of 10, but 28 of the 30 samples of 40: every one except the 0.45 and the 0.725.
The bigger samples are not more correct on average; both sizes are aimed at 0.6. What changed is how tightly they gather around it.
Day 2: how close is one estimate?
Caius has a different bag of 200 beads and does not know its share of blue. He draws one random sample of 40 and finds 23 blue. Use Figure 2 to say how close his estimate is likely to be. What could he say if he had drawn only 10 beads and found 6 blue?
SolutionHis sample proportion is \displaystyle \frac{23}{40} = 0.575.
In Figure 2, 28 of the 30 samples of 40 landed within 0.1 of the bag’s share. Caius borrows that spread, which is fair if his bag’s share is not far from the simulated 0.6, and 0.575 suggests it is not. Turned around, his bag’s share is likely within about 0.1 of 0.575:
With 10 beads and 6 blue, his estimate would be \displaystyle \frac{6}{10} = 0.6. Only 20 of the 30 samples of 10 landed within 0.1 of the truth, which is not “most” by a wide margin. To cover nearly all of them he has to go out to 0.2: from 0.4 to 0.8 lie 28 of the 30. So the honest statement is weaker: the share is probably between about 0.4 and 0.8. In this simulation, four times as many beads halved the likely distance.
Day 2: simulating samples of game cards with random digits
A game has 500 cards, and 100 of them are rare. Eamon-Lee wants to see how much the share of rare cards varies among random samples of 10 cards, without handling 500 cards. Design a simulation with random digits, run four samples of 10 from the row below, and then treat the whole row as one sample of 40.
The parameter is \displaystyle \frac{100}{500} = 0.2. Each random digit is equally likely to be any of the ten digits, so let 0 and 1 mean a rare card and 2 to 9 a common one: each digit is then rare with chance \displaystyle \frac{2}{10} = 0.2. One digit is one card, and ten digits are one sample. (Real cards are not put back, but ten out of 500 is so small a part of the pack that the chance barely moves; the ten percent condition makes this precise.)
Take the digits ten at a time and count the 0s and 1s. Sample 1 is 04480\,18774, sample 2 is 88971\,27244, sample 3 is 16014\,30837 and sample 4 is 65928\,69788:
| Sample | Rare cards | Proportion |
|---|---|---|
| 1 | 3 | 0.3 |
| 2 | 1 | 0.1 |
| 3 | 4 | 0.4 |
| 4 | 0 | 0 |
Four samples from a pack whose share is 0.2 gave 0.3, 0.1, 0.4 and 0: each off by 0.1 or 0.2, on both sides of 0.2.
All 40 digits together are one random sample of 40 cards. It holds 3 + 1 + 4 + 0 = 8 rare cards, and \displaystyle \frac{8}{40} = 0.2, exactly the parameter this time. The pooled sample is steadier than its four pieces, as Result 3 says, though another row would not always hit 0.2 exactly.
Day 2: is this result surprising?
The seller of the bead bags says every bag is 0.6 blue. Eamon-Lee draws a random sample of 40 beads from one bag and finds 16 blue. Bettina-Rose draws a random sample of 10 from another bag and finds 4 blue. Use Figure 2 to judge whether either result gives reason to doubt the seller.
SolutionBoth found \displaystyle \frac{16}{40} = \frac{4}{10} = 0.4. Figure 2 shows what a bag that really is 0.6 blue produces, so compare each result with samples of its own size.
Among the 30 samples of 10, a proportion of 0.4 or less came up 2 + 4 = 6 times: one sample in five. That is ordinary for a 0.6 bag, so Bettina-Rose’s handful gives no real reason to doubt the seller.
Among the 30 samples of 40, the lowest was 0.45; not one reached 0.4. A 0.6 bag seldom produces Eamon-Lee’s result, so it is a genuine reason to doubt that his bag is 0.6 blue.
It is not proof: a 0.6 bag can give 16 blue beads in 40, only rarely, and thirty simulated samples is a small picture. But the same proportion gets two verdicts, and sample size is the whole difference.
Boundaries
The samples must be random. A convenience sample or any other biased method centers in the wrong place, and making it bigger only tightens the pile around the wrong value. A larger sample cures spread, never bias.
A simulation needs the population, or a model of it. The bead bag could be simulated because its share was known. A real study borrows a spread from a model, as Caius did. Proportions spread most when the share is near 0.5 and less near 0 or 1, so the model’s share must be close to the real one.
Thirty simulated samples is a rough picture. Run the simulation again and the dots move, so a count like “28 of 30” is itself an estimate. Drawing more samples of 10 shows their spread more clearly without making it any smaller; only a bigger sample size does that.
Digits stand for draws with replacement. That matches real sampling only while the sample is a small part of the population. A “sample” of all 200 beads is a census, with no sampling variability at all.
Where this goes next
The pile of statistics from many samples of one size is a sampling distribution. The sampling distribution of a sample proportion and the sampling distribution of a sample mean describe those piles exactly, without simulating.
“Probably within about 0.1” grows into the margin of error in simulating a sampling distribution and a margin of error, and center and spread become the two questions asked of every estimate in bias and variability of an estimator. Judging a result against simulated chance, as Example 6 did, is the start of randomization tests and of statistical inference as a whole.
Exercises
Day 1: statistics from samples
- A random sample of 10 beads from the bag of 200 beads, 120 of them blue, holds 4 blue beads. Find the sample proportion and how far it is from the parameter.
- The game of Example 5 has 500 cards, 100 of them rare. Two random samples of 25 cards hold 3 and 7 rare cards. Find each sample proportion and how far each is from the parameter.
Day 1: reading a dot plot
- Use Figure 1. (a) How many samples gave exactly 0.6? (b) How many gave 0.4 or less? (c) What fraction of the samples landed more than 0.1 away from 0.6?
- A bakery made 60 muffins with a mean of 9 raisins each. Twenty random samples of 4 muffins gave these sample means: 8.25, 8.50, 9.50, 9.50, 9.75, 8.25, 9.00, 10.75, 9.50, 8.25, 9.50, 8.75, 11.25, 9.50, 9.25, 9.00, 9.25, 8.25, 8.00, 9.50. (a) Give the smallest and largest. (b) How many are within 0.5 of 9? (c) Find the mean of the twenty values.
Day 1: drawing samples with random digits
- Use the leaf table and the method of Example 2 to draw a sample of 5 leaves from the row 31955 \quad 45904. Find the sample mean and how far it is from 8 cm.
- Do the same with the row 81718 \quad 14632 \quad 77905, skipping any leaf already chosen.
Day 2: bigger samples
- Use Figure 2. How many samples of 10, and how many samples of 40, landed within 0.05 of 0.6?
- Twenty random samples of 20 muffins from the bakery of Exercise 4 gave these sample means: 9.10, 9.10, 9.75, 9.00, 8.95, 9.00, 9.35, 8.80, 8.85, 8.65, 8.95, 9.50, 8.90, 8.40, 8.85, 8.50, 8.55, 8.95, 8.60, 9.40. (a) Give the smallest and largest. (b) How many are within 0.5 of 9? (c) Find the mean of the twenty values. (d) Compare with Exercise 4.
Day 2: how close is an estimate?
- Aldric draws a random sample of 40 beads from a bag of 200 whose share of blue is unknown, and finds 26 blue. Using Figure 2, say in words how close his estimate is likely to be.
- Bettina-Rose wants an estimate of a bag’s share of blue that is likely to be within about 0.1 of the truth. Should she sample 10 beads or 40? Use the counts in Figure 2.
Day 2: simulating samples
- A jar of mixed beans is 0.3 black beans. Let the digits 0, 1 and 2 stand for a black bean. Split the row 31494 74614 00581 46118 91858 32236 into three samples of 10 beans and find each sample proportion of black beans.
- Treat all 30 digits of Exercise 11 as one sample of 30 beans. Find the sample proportion and how far it is from 0.3.
- For the 500 game cards, Zinnia reads digits in pairs and lets 00 to 19 stand for a rare card. Why does that give the right chance? Use the row 69143 \quad 97501 \quad 35223 \quad 72495 to run two samples of 5 cards and find each sample proportion.
- Another jar of beans is one quarter black. Describe a device, not random digits, that simulates drawing a random sample of 10 beans from it, and say what one trial is.
Reasoning
- Zinnia draws a random sample of 10 beads from the bag of Figure 1 and gets 0.9. No sample in Figure 1 did that. Does her result show that she sampled unfairly? Explain.
- Explain why one bead more or less moves a sample of 10 further than a sample of 40, and what that says about the spread of the two dot plots.
- Damaris says: “My sample of 40 was chosen at random, so its proportion must equal the bag’s.” Use Figure 2 to explain what is wrong.
- Two dot plots each show 30 sample proportions from one bag. Both center near 0.6; one runs from 0.3 to 0.9 and the other from 0.5 to 0.7. One used samples of 10 and the other samples of 50. Which is which, and why does the center not tell you?
- Every simulation in this lesson used a population whose parameter was known. Explain why that was needed, and what changes when a real study does not know its parameter.
Answers
- 0.4, which is 0.2 below the parameter: \displaystyle \frac{4}{10} = 0.4 and \displaystyle \frac{120}{200} = 0.6.
- 0.12 and 0.28, each 0.08 from the parameter \displaystyle \frac{100}{500} = 0.2, since \displaystyle \frac{3}{25} = 0.12 and \displaystyle \frac{7}{25} = 0.28.
- (a) 9. (b) 6: 2 at 0.3 and 4 at 0.4. (c) \displaystyle \frac{10}{30} = \frac{1}{3}: the 2 at 0.3, the 4 at 0.4 and the 4 at 0.8.
- (a) Smallest 8.00, largest 11.25. (b) 12 of the 20 lie from 8.5 to 9.5. (c) The values add to 183.5, so the mean is \displaystyle \frac{183.5}{20} = 9.175, about 9.2, close to the population mean of 9.
- 8.4 cm, 0.4 cm above 8 cm. The pairs 31, 95, 54, 59, 04 have lengths 10, 10, 9, 5, 8, and \displaystyle \frac{42}{5} = 8.4.
- 7.8 cm, 0.2 cm below 8 cm. The pairs are 81, 71, 81 (skipped), 46, 32, 77, with lengths 10, 7, 8, 7, 7, and \displaystyle \frac{39}{5} = 7.8.
- Within 0.05 means from 0.55 to 0.65. Samples of 10: only the 9 at 0.6. Samples of 40: 4 + 4 + 6 + 3 = 17.
- (a) Smallest 8.40, largest 9.75. (b) 18 of the 20. (c) The values add to 179.15, so the mean is 8.9575, about 9.0. (d) Both center near 9, but these stretch over 1.35 raisins against 3.25, and 18 rather than 12 land within 0.5 of 9.
- His estimate is \displaystyle \frac{26}{40} = 0.65. Since 28 of the 30 samples of 40 landed within 0.1 of the truth, the bag’s share is probably between about 0.55 and 0.75.
- 40. In Figure 2, 28 of the 30 samples of 40 landed within 0.1 of 0.6, but only 20 of the 30 samples of 10 did.
- 0.2, 0.5 and 0.3. The samples 31494\,74614, 00581\,46118 and 91858\,32236 hold 2, 5 and 3 of the digits 0, 1, 2.
- \displaystyle \frac{10}{30} = \frac{1}{3} \approx 0.33, about 0.03 above 0.3. The three samples hold 2 + 5 + 3 = 10 black beans.
- The pairs 00 to 99 are 100 equally likely outcomes and 00 to 19 is 20 of them, so the chance is \displaystyle \frac{20}{100} = 0.2, the share of rare cards. The pairs 69, 14, 39, 75, 01 hold 2 rare cards, a proportion of 0.4; the pairs 35, 22, 37, 24, 95 hold none, a proportion of 0.
- One answer: toss two coins and let two heads, which has chance \displaystyle \frac{1}{4}, stand for a black bean. One trial is ten such tosses, one per bean, and its result is the number of black beans divided by 10.
- No. A fair random sample of 10 from a bag that is 0.6 blue can hold 9 blue beads; it is just uncommon, which is why thirty samples did not show it. One surprising sample is weak evidence; more samples would show whether something is wrong.
- Each bead is \displaystyle \frac{1}{10} = 0.1 of a sample of 10 but only \displaystyle \frac{1}{40} = 0.025 of a sample of 40. Missing by 0.2 takes 0.2 \times 40 = 8 extra blue beads in a sample of 40, against 2 in a sample of 10, so large misses are rarer and the plot is narrower.
- Random does not mean exact. Only 4 of the 30 samples of 40 in Figure 2 landed on 0.6; the rest missed by up to 0.15.
- The plot from 0.5 to 0.7 used samples of 50 and the one from 0.3 to 0.9 used samples of 10, because bigger samples spread less. Random samples of every size center on the same parameter, so only the spread shows the size.
- Only with the parameter known can you see how far each simulated sample landed from the truth. A real study has one sample and an unknown parameter, so it cannot see its own miss; it borrows a spread from a simulation of a similar population and says the truth is probably within about that distance of its sample.