Algebra 2 · 7.SP.A.2 · S.IC.B.4

Simulating a sampling distribution and a margin of error

One poll of 50 people says 38% are in favor. How far from the truth could that be, and how would anyone find out?

You need

Sampling variability and simulating itMath 7

Unlocks

Comparing two treatments by re-randomizingAlgebra 2

Where you are 7 in Sampling distributions

Statistics7

Distributions8

Sampling distributions7

Bias and variability of an estimator Sampling variability and simulating it

Simulating a sampling distribution and a margin of error

The ten percent condition Sampling distribution of a sample mean ▾ Show all 7 (2 more)

When you reach for it

A study has one sample and one statistic: a poll’s share in favor, a survey’s mean. To report it honestly it needs a second number, how far the statistic is likely to sit from the parameter it estimates. Sampling variability showed that random samples scatter around the parameter, tighter for bigger samples. This lesson turns that picture into a number.

The tool is a simulated sampling distribution: take a population whose parameter you know, or a model of one, draw a great many random samples of the size the study used, compute the statistic for each, and look at the pile they make. The pile’s standard deviation measures how far one sample typically lands from the truth, and twice it gives a margin of error: the distance within which most samples, about 95 in 100, land.

Reach for it when you need to say “about how close” a sample statistic is, when you want to judge whether a result is surprising under some claim, or when you want to know how large a sample must be for a given precision.

The procedure

Model the population. A simulation can only copy a population it can describe. For a yes-or-no question that means a share p; for a measurement it means the whole list of values, or their shares. If the real parameter is unknown, use a value close to the sample’s own, or a cautious value (see Boundaries). The model has to be right for the simulation to mean anything.

Imitate one sample, exactly as the study drew it. Same size, same random method. If the population is large, each draw barely changes what is left, so each member can be simulated as an independent chance: a random digit, a spinner, or a computer’s random number. This is what the ten percent condition makes precise. One simulated sample gives one value of the statistic, called one trial. Designing a simulation covers matching a device to a chance.

Repeat, many times. Thirty trials sketch the pile; hundreds draw it well, which is why a computer does the work. More trials make the picture of the pile sharper. They do not make the pile narrower: only a larger sample size does that.

Describe the pile. Plot the trials as a dot plot or histogram, and report three things.

Its center, the mean of all the trials. For a random sample it lands near the parameter.
Its spread, the standard deviation of the trials. Here the trials are the whole set being described, so divide by their count. This number is how far a single sample typically misses.
Its shape. Most piles in this lesson form a single, nearly even hump, close to a normal curve.

Turn the spread into a margin of error. In a single, nearly even hump about 95\% of the values lie within 2 standard deviations of the center, the middle figure of the empirical rule met with z-scores and the empirical rule. So

\text{margin of error} \approx 2 \times \text{spread}

where the spread is the standard deviation of the simulated statistics, and a real sample’s result is reported as

\text{statistic} \pm \text{margin of error}

The reason the reversal is allowed: “the sample landed within 0.14 of the truth” and “the truth is within 0.14 of the sample” are the same sentence. About 95 samples in 100 land within the margin, so for about 95 samples in 100 the interval reaches the truth. Which kind your one sample is, you cannot tell.

Check against the formula. Long simulations settle on spreads that a formula predicts, proved in Sampling distribution of a sample proportion and Sampling distribution of a sample mean:

\text{proportions: } \sqrt{\frac{p(1 - p)}{n}}

\text{means: } \frac{\sigma}{\sqrt{n}}

where n is the sample size and \sigma is the population’s standard deviation. A simulation of a few hundred trials lands near the formula, not on it. Both formulas have \displaystyle \sqrt{n} underneath, which is the rule about sample size: four times the sample halves the spread, and nine times cuts it to a third.

Worked examples

Example 1

One trial with random digits

In a large town, 0.4 of adults back building a new public swimming pool. Design a simulation of a random sample of 50 adults using random digits, and run one trial on this row:

71852 \; 93367 \; 29837 \; 39765 \; 12625

32085 \; 62323 \; 60579 \; 58592 \; 17126

Solution

Each random digit is equally likely to be any of 0 to 9. Let the four digits 0, 1, 2, 3 stand for an adult who backs the pool and 4 to 9 for one who does not; each digit then says “yes” with chance \displaystyle \frac{4}{10} = 0.4, the town’s share. The town is large, so treating each adult as a fresh, independent chance is fair.

One digit is one adult and 50 digits are one sample. Counting the digits 0 to 3 group by group gives 2, 2, 2, 1, 3, then 3, 4, 1, 1, 3:

2 + 2 + 2 + 1 + 3 + 3 + 4 + 1 + 1 + 3 = 22

So this trial’s sample proportion is \displaystyle \frac{22}{50} = 0.44.

That is one dot. A sampling distribution needs hundreds of dots, each 50 digits long, which is tens of thousands of digits: the reason the next examples hand the drawing to a computer, which does the same thing with random numbers.

Example 2

Reading a computer run of 200 samples

A computer ran 200 trials of the simulation in Example 1. Figure 1 shows them; the table counts them.

Figure 1One computer run of 200 simulated random samples of 50 adults from a town where 0.4 back the pool (the triangle). Each dot is one sample’s proportion. The bracket is 0.4 \pm 0.14. A second run would move the dots a little.

Value	Count	Value	Count
0.24	3	0.44	19
0.26	2	0.46	15
0.28	3	0.48	17
0.30	7	0.50	9
0.32	12	0.52	5
0.34	19	0.54	5
0.36	14	0.56	3
0.38	22	0.58	2
0.40	21	0.60	1
0.42	21

Describe the center, spread and shape. Compare the spread with the formula, and count how many samples landed within 2 standard deviations of 0.4.

Solution

Center. All 200 samples together held 4088 “yes” answers among 200 \times 50 = 10000 adults. Every sample has the same size, so the mean of the 200 proportions is

\frac{4088}{10000} = 0.4088

close to the town’s 0.4, as a random sample should be.

Spread. A calculator gives the standard deviation of the 200 values as about 0.072. The formula for proportions predicts

\sqrt{\frac{0.4 \times 0.6}{50}} = \sqrt{0.0048} \approx 0.069

so this run sits a little above the formula’s value, as a run of 200 trials may.

Shape. A single hump, highest from 0.38 to 0.42 and thinning out evenly on both sides.

Within 2 standard deviations. Twice the spread is 2 \times 0.072 = 0.144, about 0.14. The band 0.4 \pm 0.14 runs from 0.26 to 0.54, the bracket in Figure 1. Outside it lie only 3 samples at 0.24, and 3, 2 and 1 at 0.56, 0.58 and 0.60, which is 3 + 3 + 2 + 1 = 9 samples. So 191 of the 200, or 95.5\%, landed within 0.14 of the truth: the “about 95 in 100” the procedure promised. This is one run’s count; a second run would give a number near it, not the same number.

Example 3

From the spread to a margin of error

Casimira polls a random sample of 50 adults in a different large town, whose share in favor of a pool is unknown, and 19 say yes. Give her estimate with a margin of error, and say whether the poll shows that most adults there are against the pool.

Solution

Her sample proportion is \displaystyle \frac{19}{50} = 0.38.

She cannot simulate her own town, because its share is the thing she does not know. She borrows the spread from a model whose share is near her result. Example 2 used a share of 0.4, close to 0.38, and the spread hardly changes between nearby shares: the formula with 0.38 gives

\sqrt{\frac{0.38 \times 0.62}{50}} \approx 0.069

the same as for 0.4 to three places. So her margin of error is about 0.14, and she reports

0.38 \pm 0.14, \quad \text{from } 0.24 \text{ to } 0.52

The honest reading: if this town’s true share is anywhere in that range, a sample like hers is ordinary, and a method like hers catches the truth about 95 times in 100.

Does the poll show a majority against? No. The range reaches 0.52, above one half, so a town with a small majority in favor could easily have produced her sample. A sample of 50 is too small to settle a question this close.

Example 4

How sample size changes the spread

Figure 2 adds 200 simulated samples of 200 adults from the same town (share 0.4). Their mean is 0.39985 and their standard deviation is about 0.038. Compare them with the samples of 50, find a margin of error for samples of 200, and find the sample size that would give a margin of error of 0.05.

Figure 2One simulated run of each sample size from the same town. Both piles center near 0.4; the samples of 200 gather about twice as tightly.

Solution

Center. Both piles center near 0.4 (0.4088 and 0.39985). A bigger sample does not move the aim.

Spread. Four times the sample size took the simulated spread from 0.072 to 0.038, a little more than half. The formula says exactly half, because

\sqrt{\frac{0.4 \times 0.6}{200}} = \sqrt{0.0012} \approx 0.035

and \displaystyle \sqrt{4} = 2 sits in the denominator: \displaystyle \frac{0.069}{2} \approx 0.035. This run came out a little wide, as Example 2’s did.

Margin of error for samples of 200. Twice the formula’s spread is about 0.07. In this run, 190 of the 200 samples of 200 landed within 0.07 of 0.4, which is 95\%. (Twice the simulated spread is 0.076, about 0.08. Either way the margin is roughly half the 0.14 for samples of 50.)

The size for a margin of 0.05. Set twice the spread equal to 0.05 and solve for n:

\begin{aligned} 2\sqrt{\frac{0.24}{n}} &= 0.05 \\ \sqrt{\frac{0.24}{n}} &= 0.025 \\ \frac{0.24}{n} &= 0.000625 \\ n &= 384 \end{aligned}

Dividing by 2, then squaring both sides (both are positive, so nothing is gained or lost), then solving the last line for n. A poll of about 384 adults gives a margin of about 0.05 when the share is near 0.4.

Example 5

A sample mean: people in each car

In another large town, a traffic count of every car on a main road gave the shares in Figure 3: 40\% of cars carry 1 person, 30\% carry 2, 15\% carry 3, 10\% carry 4 and 5\% carry 5.

Figure 3The population of cars. Its mean is 2.1 people, and its shape is lopsided, with a long tail to the right.

Find the population mean and standard deviation. A computer then drew 200 random samples of 25 cars and recorded each sample’s mean (Figure 4). The 200 sample means had a mean of 2.0938 and a standard deviation of about 0.233, and 196 of them lay within 0.47 of 2.1. Compare with the formula, and give a margin of error for the mean of 25 cars.

Figure 4One simulated run of 200 random samples of 25 cars, one dot per sample mean. The population was lopsided; the pile of means is a single, nearly even hump.

Solution

The population. Each share is the weight of its value, so the mean is

\begin{aligned} &1(0.4) + 2(0.3) + 3(0.15) \\ &\quad + 4(0.1) + 5(0.05) = 2.1 \text{ people} \end{aligned}

For the standard deviation, the mean of the squared values is

\begin{aligned} &1(0.4) + 4(0.3) + 9(0.15) \\ &\quad + 16(0.1) + 25(0.05) = 5.8 \end{aligned}

so the variance is 5.8 - 2.1^2 = 1.39 and

\sigma = \sqrt{1.39} \approx 1.179

The simulated means. Their center, 2.0938, is close to 2.1. Their spread, 0.233, is close to the formula’s

\frac{\sigma}{\sqrt{n}} = \frac{1.179}{\sqrt{25}} = \frac{1.179}{5} \approx 0.236

A single car varies by more than a person on average; a mean of 25 cars varies by about a quarter of one.

Shape. The population is lopsided, piled at 1 with a tail to the right. The sample means are not: they form a single, nearly even hump, with only a faint trace of the right tail (two stray means at 2.88 and 3.0). Averaging pulls lopsided data toward a hump, an effect the central limit theorem describes.

Margin of error. Twice the spread is 2 \times 0.233 \approx 0.47. In this run 196 of the 200 means, 98\%, lay within 0.47 of 2.1; that is a little more than 95 in 100, which one run of 200 can easily give. So the mean number of people in a random sample of 25 cars is likely within about 0.47 of the true mean.

If a survey of 25 random cars on another road found a mean of 1.8 people, it would report 1.8 \pm 0.47, from 1.33 to 2.27 people per car.

Example 6

Is a claim believable?

A campaign leaflet says 40\% of the adults in the town of Examples 1 and 2 back the pool. Lorcan’s random sample of 50 adults finds only 13 in favor. Use the simulation in Example 2 to judge the leaflet’s claim.

Solution

Lorcan’s proportion is \displaystyle \frac{13}{50} = 0.26. Figure 1 shows what samples of 50 look like when the share really is 0.4. In the table, a proportion of 0.26 or less came up 3 + 2 = 5 times in 200: one sample in 40.

So if the leaflet is right, a result as low as Lorcan’s is rare. That is a reason to doubt the claim, not a proof against it: a town at 0.4 does sometimes give 13 in 50, and five dots out of 200 is a rough count. The same thinking, comparing one real result with a pile simulated under a claim, is the engine of randomization tests, and Comparing two treatments by re-randomizing uses it to judge experiments.

Boundaries

The simulation is only as good as its model. It needs the parameter, or a stand-in close to it. The spread of a proportion, \displaystyle \sqrt{p(1 - p)/n}, is largest at p = 0.5, so using 0.5 when nothing is known never understates the margin.

The margin of error measures chance, not bias. It describes how random samples scatter. A convenience sample, a leading question or people who choose not to answer can push every sample the same way, and no margin of error covers that. A larger sample only shrinks the scatter around the wrong place.

Independent draws need a large population. Simulating each member as a fresh chance is fair only while the sample is a small part of the population (the ten percent condition). A “sample” of the whole population has no sampling variability at all.

“Twice the spread covers about 95%” needs a hump. It holds for a single, nearly even pile. A small sample from a share near 0 or 1 gives a lopsided pile (most samples of 50 from a share of 0.02 hold 0 or 1 yes), and there the rule fails; a simulation shows it plainly.

A simulated spread is itself an estimate. Two hundred trials gave 0.072 against the formula’s 0.069. More trials bring the simulated spread closer to the formula; they never shrink the spread itself.

“About 95 in 100” describes the method. One interval either contains the parameter or does not; the 95 in 100 is how often the method succeeds over many samples.

Where this goes next

The pile simulated here is a sampling distribution. The formulas it was checked against are proved in Sampling distribution of a sample proportion and Sampling distribution of a sample mean, and the hump that means form even from lopsided data is the central limit theorem. “Statistic plus or minus a margin” becomes point estimates and the margin of error and then Confidence intervals, where the spread is estimated from the sample itself, the standard error, instead of being simulated.

Example 6’s question, “is this result unusual if the claim is true?”, is the start of randomization tests. The next lesson, Comparing two treatments by re-randomizing, simulates by reshuffling an experiment’s results to ask whether a difference between two treatments is bigger than chance alone would make.

Exercises

Setting up a simulation

In a large town, 0.4 of adults back the pool. Let the digits 0 to 3 stand for “yes” and run one trial of a sample of 20 adults on the row 61794 \; 95642 \; 26641 \; 64854. Give the sample proportion.
In a large school, 0.25 of students cycle to school. Describe how to simulate one random sample of 40 students with pairs of random digits, and say what one trial records.

Reading the simulation

Exercises 3 to 5 use the table of Example 2.

How many of the 200 samples gave exactly 0.40? Which proportion came up most often?
How many samples landed within 0.1 of 0.4? What fraction of the 200 is that?
In how many samples did half or more of the adults back the pool? What fraction of the 200 is that, and what does it say about using one sample of 50 to decide whether the pool has majority support?

Spread and margin of error

Find \displaystyle \sqrt{\frac{0.4 \times 0.6}{50}} to three decimal places and the margin of error it gives.
A share of 0.5 and samples of 100: find the spread of the sample proportions and the margin of error.
A share of 0.1 and samples of 50: find the spread of the sample proportions to three decimal places and the margin of error.
A random sample of 50 adults finds 27 in favor. Using a margin of error of 0.14, give the interval. Does the poll show majority support?
A random sample of 200 adults finds 92 in favor. Using a margin of error of 0.07, give the interval.

Sample size

Find the spread of the sample proportions for a share of 0.4 and samples of 800, and the margin of error. Compare with samples of 200.
How large a sample gives a margin of error of 0.04 when the share is near 0.4?
With nothing known about the share, a pollster uses p = 0.5. How large a sample gives a margin of error of 0.03?
By what factor must a sample size grow to halve the margin of error? To cut it to a third?

Sample means

For the cars of Example 5 (\sigma \approx 1.179 people), find the spread of the mean of 100 cars and the margin of error.
A machine fills bags of dried lentils with a mean of 500 g and a standard deviation of 15 g. Find the spread of the mean mass of 9 bags and of 36 bags, and the margin of error for each.
A survey of 25 random cars on a road finds a mean of 2.36 people. Using the margin of error of Example 5, give the interval. Is a true mean of 2.1 ruled out?

Reasoning

Casimira reruns the simulation of Example 2 with 2000 trials instead of 200. Will the pile of sample proportions be narrower? Explain.
A website asks its readers to vote on the pool, and 5000 do. Explain why a margin of error of about 0.014 says little about the town’s true share.
Explain why using p = 0.5 when the share is unknown can never give too small a margin of error.
The 200 simulated samples of 50 centered at 0.4088, not 0.4. Explain why, and whether it means the simulation was biased.
A share of 0.02 and samples of 50. Find the formula’s spread to three decimal places, and explain why “the share \pm twice the spread” makes no sense here.

Answers

0.2. The groups 61794, 95642, 26641, 64854 hold 1, 1, 2 and 0 of the digits 0 to 3, so 4 yes in 20, and \displaystyle \frac{4}{20} = 0.2.
Read digits in pairs, 00 to 99, all equally likely. Let 00 to 24, which is 25 of the 100 pairs, stand for a cyclist, a chance of 0.25. One trial is 40 pairs, one per student, and records the number of cyclists divided by 40.
21 samples gave 0.40. The most common proportion was 0.38, with 22 samples.
From 0.30 to 0.50: 7 + 12 + 19 + 14 + 22 + 21 + 21 + 19 + 15 + 17 + 9 = 176 samples, \displaystyle \frac{176}{200} = 0.88.
9 + 5 + 5 + 3 + 2 + 1 = 25 samples, \displaystyle \frac{25}{200} = \frac{1}{8}. Even though only 0.4 of the town backs the pool, about one sample of 50 in eight shows half or more in favor, so one sample of 50 cannot reliably settle a majority question near this share.
\displaystyle \sqrt{0.0048} \approx 0.069, and the margin of error is 2 \times 0.069 \approx 0.14.
\displaystyle \sqrt{\frac{0.5 \times 0.5}{100}} = \sqrt{0.0025} = 0.05, so the margin of error is 0.1.
\displaystyle \sqrt{\frac{0.1 \times 0.9}{50}} = \sqrt{0.0018} \approx 0.042, so the margin of error is about 0.085.
\displaystyle \frac{27}{50} = 0.54, so 0.54 \pm 0.14, from 0.40 to 0.68. No: the interval reaches below 0.5, so a town without a majority could have given this sample.
\displaystyle \frac{92}{200} = 0.46, so 0.46 \pm 0.07, from 0.39 to 0.53.
\displaystyle \sqrt{\frac{0.24}{800}} = \sqrt{0.0003} \approx 0.017, margin about 0.035. Both are half the values for samples of 200 (spread 0.035, margin 0.07), because 800 is four times 200.
n = 600: \displaystyle 2\sqrt{\frac{0.24}{n}} = 0.04 gives \displaystyle \frac{0.24}{n} = 0.0004.
1112: \displaystyle 2\sqrt{\frac{0.25}{n}} = 0.03 gives \displaystyle \frac{0.25}{n} = 0.000225, so n \approx 1111.1, and rounding up keeps the margin at or below 0.03.
By 4 to halve it and by 9 to cut it to a third, because the spread has \displaystyle \sqrt{n} in its denominator and \displaystyle \sqrt{4} = 2, \displaystyle \sqrt{9} = 3.
\displaystyle \frac{1.179}{\sqrt{100}} = \frac{1.179}{10} \approx 0.118 people, margin about 0.24.
\displaystyle \frac{15}{\sqrt{9}} = 5 g with margin 10 g; \displaystyle \frac{15}{\sqrt{36}} = 2.5 g with margin 5 g.
2.36 \pm 0.47, from 1.89 to 2.83 people. No: 2.1 is inside, so a road with the town’s mean could have given this sample.
No. More trials draw the same pile in more detail, so its spread settles closer to the formula’s 0.069. The spread is set by the sample size, 50, and only a larger sample would narrow it.
The readers chose themselves, so the sample is not random: people with strong views, or with a reason to visit the website, can push every such poll the same way. The margin of error, \displaystyle 2\sqrt{\frac{0.25}{5000}} \approx 0.014, measures only chance scatter, and with bias present a large sample is precise about the wrong value.
p(1 - p) = 0.25 - (p - 0.5)^2, which is never more than 0.25 and equals it only at p = 0.5. So \displaystyle \sqrt{\frac{p(1-p)}{n}} is largest at p = 0.5, and using it gives a margin at least as large as the true one.
A simulation draws only 200 of the possible samples, so its center lands near the parameter, not on it; 0.4088 is about 0.009 off, a small fraction of the 0.072 spread. A biased method would sit off-center in run after run; a second run of this one would land on the other side of 0.4 about as often.
\displaystyle \sqrt{\frac{0.02 \times 0.98}{50}} = \sqrt{0.000392} \approx 0.020. Twice that is 0.04, and 0.02 - 0.04 = -0.02, a negative proportion. With only 1 yes expected in 50, the pile is stacked at 0 and 0.02 with a tail to the right, not an even hump, so the 95 in 100 rule does not apply.