Interval for a difference of proportions
Two groups have different shares of some feature in their samples. How far apart are the shares in the two whole populations, or, in an experiment, how much did one treatment change the share compared with the other?
Where you are 5 in Inference for proportions
When you reach for it
Maartje works for a regional energy office and wants to compare two Dutch cities: what share of households in Utrecht have solar panels on their roof, and what share in Groningen? She cannot visit every household, so she takes an independent random sample in each city: 156 of 400 Utrecht households have panels, and 117 of 450 Groningen households do (made-up data throughout this lesson). The sample shares are 0.39 and 0.26, a gap of 0.13. A different pair of samples would give a different gap, so the question is what range of gaps between the two whole cities this pair supports.
That is the job of the two-sample interval for a difference of proportions. It is the move whenever a yes-or-no feature is compared across two groups: two populations sampled separately, or two treatment groups in an experiment. It is the One sample interval for a proportion and sample size with two samples in place of one: the same plan of naming the parameter, checking conditions, computing and concluding, with a standard error built for a difference. The lesson takes two days: Day 1 compares two populations from two random samples, and Day 2 compares two treatments in a randomized experiment.
The procedure
Day 1: two independent random samples
The Sampling distribution of a difference lesson worked out how \hat{p}_1 - \hat{p}_2 behaves over repeated pairs of independent random samples: it centers at p_1 - p_2, its variance is the sum of the two variances, and with enough successes and failures in both samples its shape is close to normal. An interval follows the usual pattern, estimate plus or minus a Critical value times a Standard error. The true standard deviation contains the unknown p_1 and p_2, so the sample shares stand in for them:
The square root is \text{SE}_{\hat{p}_1 - \hat{p}_2}, and z^* times it is the Point estimates and the margin of error.
Name the parameters. p_1 and p_2 are the shares of two stated populations with a stated feature, and the interval is about their difference p_1 - p_2. Saying which group is 1 fixes the order of subtraction, and the sign of every number that follows depends on it.
Check the conditions for each sample. The Conditions for one proportion apply to both samples, plus one more:
- Random and independent. Each sample is random from its own population, and the two samples are drawn independently, so neither sample’s members say anything about the other’s. The variance of a difference is the sum of the variances only for independent quantities.
- Ten percent. Each sample is at most a tenth of its population, so the draws within it are close to independent.
- Large counts. n_1\hat{p}_1, n_1(1 - \hat{p}_1), n_2\hat{p}_2 and n_2(1 - \hat{p}_2) are all at least 10, so the normal model for the difference is a good one. These are simply the four counts of successes and failures.
When all three hold, the procedure is called a two-sample z interval for p_1 - p_2.
Compute and conclude in context. The conclusion names the level, the two ends, both populations and the feature, and says which is larger: “We are 95\% confident that the share of ... in population 1 is between ... and ... higher than in population 2.” What the level promises is the subject of Confidence level and interpreting an interval.
Where 0 sits. A difference of 0 means the two shares are equal. If the whole interval lies above 0, the data are evidence, at that level, that p_1 > p_2; if it lies below, that p_1 < p_2. If it contains 0, equal shares are plausible, and so are differences of either sign: the data give no evidence of a difference, and no evidence that there is none.
Technology. A calculator’s 2-PropZInt takes x_1, n_1, x_2, n_2 and the level, and prints the interval for p_1 - p_2 with \hat{p}_1 and \hat{p}_2. It subtracts in the order entered and checks no condition.
Day 2: two treatments in a randomized experiment
In an experiment the two groups are not sampled from two populations. One set of subjects is split at random into two treatment groups, as in a Completely randomized design and replication, and p_1 and p_2 are the shares that would show the response if all the subjects received treatment 1, or all received treatment 2. The same formula estimates p_1 - p_2, and the conditions change in one place each:
- Random. The random condition is Random assignment and confounding of subjects to treatments. Chance in which subjects land in which group is what makes \hat{p}_1 - \hat{p}_2 vary, and the standard error formula describes that variation closely when the counts are large.
- Ten percent. Nothing is sampled from a larger population, so this condition is not checked. It returns only if the subjects were themselves a random sample of some population.
- Large counts. Unchanged: the four counts of successes and failures in the two groups are each at least 10.
The conclusion is about cause. Random assignment makes the two groups alike, apart from chance, in everything except the treatment. So an interval lying wholly on one side of 0 is evidence that the treatment caused the difference, which an observational comparison of two populations can never show (Observational studies and experiments). The cause applies to the subjects in the experiment. Extending it to a wider population needs the subjects to be a random sample of that population.
The order of subtraction. Writing p_2 - p_1 instead of p_1 - p_2 changes the sign of the estimate and swaps and negates the two ends: an interval from a to b becomes one from -b to -a. The width, the standard error and the conclusion stay the same. Only the wording follows the sign: a negative interval for p_1 - p_2 says group 2 has the larger share.
Worked examples
Day 1: solar panels in two cities
Maartje’s samples are independent simple random samples of households in each city, and, as part of the problem, take each city to have more than 10{,}000 households. In Utrecht 156 of 400 households have rooftop solar panels; in Groningen 117 of 450 do. Construct and interpret a 95\% confidence interval for the difference in the shares.
SolutionParameters. Let p_U be the proportion of all Utrecht households with solar panels and p_G the proportion of all Groningen households. The interval is for p_U - p_G.
Conditions. Each sample was drawn at random from its own city, and the two were drawn separately, so they are random and independent. A tenth of 10{,}000 is 1{,}000, and both 400 and 450 are below it. The counts are 156 and 400 - 156 = 244 in Utrecht, and 117 and 450 - 117 = 333 in Groningen, all at least 10. All conditions hold.
Compute. The shares are \hat{p}_U = 156/400 = 0.39 and \hat{p}_G = 117/450 = 0.26, and 95\% gives z^* = 1.96.
The interval is 0.13 \pm 0.0627, from 0.0673 to 0.1927, or about 0.067 to 0.193.
Conclude. We are 95\% confident that the share of Utrecht households with solar panels is between 6.7 and 19.3 percentage points higher than the share of Groningen households. The whole interval lies above 0, so the samples are evidence that the share really is higher in Utrecht. The comparison is of two populations, so it says nothing about why.
Day 1: reading 2-PropZInt output
Wouter writes the office’s yearly summary, which uses 90\% intervals. He enters x_1 = 156, n_1 = 400, x_2 = 117, n_2 = 450 and level 0.90 into 2-PropZInt.
| 2-PropZInt | |
|---|---|
| interval | (0.0774,\ 0.1826) |
| \hat{p}_1 | 0.39 |
| \hat{p}_2 | 0.26 |
| n_1, n_2 | 400, 450 |
Find the estimate and the margin from the output, confirm the margin by hand, and state the conclusion.
SolutionThe interval is centered at \hat{p}_1 - \hat{p}_2, so the center is the midpoint of the two ends and the margin is half the width:
At 90\% the critical value is z^* = 1.645, and with the standard error from the previous example, 1.645 \times 0.03197 \approx 0.0526. The two agree. The calculator subtracted in the order entered, Utrecht minus Groningen, so positive values favor Utrecht. We are 90\% confident that Utrecht’s share of households with solar panels is between 7.7 and 18.3 percentage points higher than Groningen’s. The interval is narrower than the 95\% one because z^* is smaller; the conditions are the same ones already checked.
Day 1: an interval that contains 0
The same two samples also record whether each household heats with a heat pump: 52 of the 400 Utrecht households and 63 of the 450 Groningen households do. Construct a 95\% interval for p_U - p_G for heat pumps and compare it with the solar interval.
The samples are the same random, independent samples, so only the counts need checking: 52, 348, 63 and 387 are all at least 10. The shares are \hat{p}_U = 52/400 = 0.13 and \hat{p}_G = 63/450 = 0.14.
The interval is -0.01 \pm 0.0460, from -0.0560 to 0.0360. Figure 1 sets it beside the solar interval. The solar interval lies entirely to the right of 0; this one straddles it. Plausible values run from Groningen ahead by 5.6 points to Utrecht ahead by 3.6 points, with 0 among them. The samples give no evidence that the heat pump shares differ. They do not show the shares are equal either: a real gap of 3 points in either direction is also a plausible value.
Day 2: two reminder letters
Sjoerd runs a subsidy for home solar panels in Groningen. He takes 600 owner-occupied households from one district’s register that have not applied, and assigns 300 at random to receive letter A, a plain reminder of the subsidy, and the other 300 to receive letter B, the same reminder plus how many homes on their street already have panels. Within three months, 63 of the letter A households and 87 of the letter B households apply. Construct and interpret a 95\% interval for the difference in application rates.
SolutionParameters. Let p_B be the share of these 600 households that would apply if all received letter B, and p_A the share if all received letter A. The interval is for p_B - p_A.
Conditions. The letters were assigned at random, which is the random condition for an experiment; the groups come from one random split, not from separate samples. No population was sampled, so there is no ten percent check. The counts are 87 and 213 for letter B and 63 and 237 for letter A, all at least 10.
Compute. \hat{p}_B = 87/300 = 0.29 and \hat{p}_A = 63/300 = 0.21.
The interval is 0.08 \pm 0.0690, from 0.011 to 0.149.
Conclude. We are 95\% confident that, for these 600 households, sending letter B instead of letter A raises the application rate by between 1.1 and 14.9 percentage points. The interval lies above 0 and the letters were randomly assigned, so this is evidence that letter B causes more applications among these households. The households were taken from one district’s register, not sampled at random from Groningen, so the result reaches other households only as far as they resemble these.
Day 2: the other order, and a higher level
Maartje rewrites Sjoerd’s result as p_A - p_B. Find that interval, and find the 99\% interval for p_B - p_A. Does either change the conclusion?
Reversing the order negates the estimate, \hat{p}_A - \hat{p}_B = -0.08, and leaves the standard error alone, since each term under the root is unchanged. The interval is -0.08 \pm 0.0690, from -0.149 to -0.011: the old interval flipped across 0, as Figure 2 shows. It lies wholly below 0, so letter A’s rate is lower, by 1.1 to 14.9 points. The sentence is the same fact told from the other side.
The level is another matter. At 99\%, z^* = 2.576:
so the interval for p_B - p_A is 0.08 \pm 0.0907, from -0.0107 to 0.1707. It now contains 0. At 99\% confidence the experiment does not rule out that the letters work equally well. A higher level buys a wider interval, and this one is wide enough to reach 0; whether letter B helps is answered at 95\% and left open at 99\%. The level should be chosen before the data are seen, never adjusted to get the wanted answer.
Day 2: same arithmetic, no cause
Wouter compares households that attended a solar information evening in Utrecht with those that did not. From the list of about 3{,}000 attending households he takes a random sample of 200, and 70 have since installed panels. From the city’s other households, taken here to number more than 10{,}000, he takes an independent random sample of 300, and 60 have installed panels. Construct a 95\% interval for p_{\text{att}} - p_{\text{non}} and say whether it shows that the evenings cause installations.
SolutionConditions. Two independent random samples. The ten percent check: 200 \le 300, a tenth of the 3{,}000 attendees, and 300 is below 1{,}000, a tenth of the more than 10{,}000 other households. Counts 70, 130, 60 and 240 are all at least 10.
Compute. \hat{p}_{\text{att}} = 70/200 = 0.35 and \hat{p}_{\text{non}} = 60/300 = 0.20.
The interval is 0.15 \pm 0.0801, from 0.0699 to 0.2301.
Conclude. We are 95\% confident that the share of attending households that installed panels is between 7.0 and 23.0 percentage points higher than among other Utrecht households. This is evidence of a real difference between two populations, but not evidence that the evenings caused it. The households chose whether to attend, and a household already planning to install has every reason to go. To learn whether an evening changes behavior, invitations would have to be assigned at random, as Sjoerd’s letters were. The formula is identical in both studies; what the interval can claim comes from how the groups were formed.
Boundaries
- The two samples must be independent. Paired data, such as the same households asked twice, or husbands and wives from the same homes, make \hat{p}_1 and \hat{p}_2 move together, and the sum-of-variances standard error is then wrong.
- Each group needs randomness: a random sample from its own population, or random assignment to treatments. Volunteers, or groups that chose themselves, give numbers with no stated level behind them.
- The ten percent condition applies to sampling and is checked for each sample separately. For an experiment with no sampling it does not arise.
- The normal model needs at least 10 successes and 10 failures in each group. With small counts the real capture rate can fall short of the stated level.
- A cause can be claimed only from random assignment, and only for the subjects in the experiment unless they were sampled at random from a population. Two random samples of populations show a difference, never its cause.
- An interval that contains 0 is not evidence that the shares are equal; it means the data cannot tell the sign of the difference at that level.
- The margin covers chance variation only, not nonresponse, a slanted question or a list that leaves households out.
Where this goes next
The same samples can be asked a yes-or-no question instead: is the difference real? That is the Test for a difference of proportions, which starts from p_1 - p_2 = 0 and ends in Deciding and concluding from a p-value. A 95\% interval that excludes 0 and a two-sided test at the 0.05 level usually agree, though the test computes its standard error with one pooled share, because it assumes the two shares are equal. For averages rather than shares, the Interval for a difference of means keeps the same shape of estimate plus or minus a critical value times a standard error, with t^* in place of z^*.
Exercises
Use z^* = 1.645, 1.96 and 2.576 for 90\%, 95\% and 99\% confidence. All data are made up. Unless told otherwise, samples are independent random samples from cities taken to have more than 10{,}000 households each.
Day 1: two independent random samples
- In Utrecht 120 of 500 sampled households have a home charger for an electric car; in Groningen 60 of 400 do. Name the parameters, check the conditions, and construct and interpret a 95\% interval for p_U - p_G.
- In Utrecht 66 of 300 sampled households own a cargo bike; in Groningen 45 of 300 do. Construct and interpret a 90\% interval for p_U - p_G.
- In Utrecht 210 of 600 sampled households compost their food waste; in Groningen 230 of 500 do. Construct a 99\% interval for p_U - p_G and say which city has the larger share.
- For each plan, say whether the two-sample z interval may be used, and if not, which condition fails. (a) Random samples of 400 in Utrecht and 380 in Groningen, with 6 and 14 households owning a home battery. (b) A random sample of 80 of the 500 households in a village, compared with a random sample of 300 Groningen households. (c) Households who answered a poll on each city’s newspaper website. (d) Random samples of 250 in each city, with 40 and 55 successes.
- A 2-PropZInt output reads: interval (0.0200,\ 0.1400), \hat{p}_1 = 0.42, \hat{p}_2 = 0.34, n_1 = n_2 = 500. Find the two counts of successes, the margin of error, and the confidence level used.
- For Maartje’s solar samples, compute the 95\% margin of the one-sample interval for each city separately. Compare their sum with the margin 0.0627 of the difference, and say why the difference’s margin is smaller.
- A report gives a 95\% interval for p_U - p_G from 0.035 to 0.145. Find the estimate and the margin, and say whether 0 and 0.15 are plausible values.
Day 2: randomized experiments
- A subsidy website shows 1{,}000 visiting Utrecht households, at random, one of two pages: 500 see a page with a savings calculator and 140 of them apply; 500 see a plain page and 105 apply. Construct and interpret a 95\% interval for p_{\text{calc}} - p_{\text{plain}}.
- In Exercise 8, construct the interval for p_{\text{plain}} - p_{\text{calc}} and state its conclusion.
- Of 300 households in a trial, 150 are assigned at random to a home visit from an energy coach and 150 to a leaflet. Of the coached households 72 lower their thermostat setting, against 54 of the leaflet households. Construct 90\% and 99\% intervals for p_{\text{coach}} - p_{\text{leaflet}}, and say what each shows.
- In a small trial, 40 households are assigned at random to each of two letters, and 6 and 15 apply for the subsidy. Can the two-sample z interval be used? Explain.
- For each study, say whether a 95\% interval lying above 0 would show cause, and why. (a) Letters are assigned at random to households. (b) Random samples of homeowners and of renters are compared. (c) Households that asked for a free energy scan are compared with a random sample of those that did not.
- An experiment reports a 95\% interval for p_A - p_B from -0.09 to -0.02. Which treatment had the higher rate, and by how much?
Reasoning
- Explain why the standard error adds the two quantities \hat{p}_1(1 - \hat{p}_1)/n_1 and \hat{p}_2(1 - \hat{p}_2)/n_2 under one square root, rather than adding the two separate standard errors.
- The heat pump interval contained 0. Explain why Wouter may not write that “Utrecht and Groningen have the same share of heat pumps.”
- Explain why the ten percent condition is checked for Maartje’s city samples but not for Sjoerd’s letter experiment, and what plays the role of random sampling in the experiment.
- Sjoerd’s experiment found letter B better for 600 households from one district. Explain what would have to change for the conclusion to cover all Groningen households who have not applied.
- Explain why reversing the order of subtraction cannot change whether an interval contains 0.
Answers
- p_U, p_G: shares of all households in each city with an electric car charger. Random, independent; 500 and 400 are below 1{,}000, a tenth of 10{,}000; counts 120, 380, 60, 340 are at least 10. \hat{p}_U = 0.24, \hat{p}_G = 0.15, estimate 0.09, \text{SE}^2 = 0.24 \times 0.76/500 plus 0.15 \times 0.85/400, so \text{SE} \approx 0.0261, margin \approx 0.0512, interval 0.0388 to 0.1412. We are 95\% confident Utrecht’s share is 3.9 to 14.1 points higher.
- \hat{p}_U = 0.22, \hat{p}_G = 0.15, estimate 0.07. \displaystyle \text{SE} = \sqrt{(0.1716 + 0.1275)/300} \approx 0.0316, margin 1.645 \times 0.03158 \approx 0.0519, interval 0.0181 to 0.1219. Counts 66, 234, 45, 255. We are 90\% confident Utrecht’s share of cargo bike owners is 1.8 to 12.2 points higher.
- \hat{p}_U = 0.35, \hat{p}_G = 0.46, estimate -0.11. \text{SE}^2 = 0.35 \times 0.65/600 plus 0.46 \times 0.54/500, so \text{SE} \approx 0.0296, margin \approx 0.0762, interval -0.1862 to -0.0338. Counts 210, 390, 230, 270. The whole interval is below 0: Groningen’s share is larger, by 3.4 to 18.6 points.
- (a) No: only 6 successes in Utrecht. (b) No: 80 is more than a tenth of 500, which is 50. (c) No: the respondents chose themselves. (d) Yes: counts 40, 210, 55, 195 are all at least 10, and the samples are random and small.
- x_1 = 0.42 \times 500 = 210, x_2 = 0.34 \times 500 = 170. Margin \displaystyle \frac{0.1400 - 0.0200}{2} = 0.0600. \displaystyle \text{SE} = \sqrt{(0.2436 + 0.2244)/500} \approx 0.03059, so z^* \approx 0.0600/0.03059 \approx 1.96: a 95\% level.
- Utrecht: \displaystyle 1.96\sqrt{0.39 \times 0.61/400} \approx 0.0478. Groningen: \displaystyle 1.96\sqrt{0.26 \times 0.74/450} \approx 0.0405. Sum \approx 0.0883, more than 0.0627. Variances add, not standard deviations, and \displaystyle \sqrt{a^2 + b^2} < a + b for positive a and b: the two chance errors are unlikely to be large in opposite directions at once.
- Estimate \displaystyle \frac{0.035 + 0.145}{2} = 0.09, margin \displaystyle \frac{0.145 - 0.035}{2} = 0.055. 0 lies below the interval, so it is not plausible; 0.15 lies above it, so it is not plausible either.
- Random assignment; counts 140, 360, 105, 395. \hat{p}_{\text{calc}} = 0.28, \hat{p}_{\text{plain}} = 0.21, estimate 0.07, \displaystyle \text{SE} = \sqrt{(0.2016 + 0.1659)/500} \approx 0.0271, margin \approx 0.0531, interval 0.0169 to 0.1231. We are 95\% confident the calculator page raises the application rate by 1.7 to 12.3 points for these households; random assignment makes this evidence of cause.
- From -0.1231 to -0.0169: the plain page’s rate is lower by 1.7 to 12.3 points. Same conclusion, same width.
- \hat{p}_{\text{coach}} = 0.48, \hat{p}_{\text{leaflet}} = 0.36, estimate 0.12, \displaystyle \text{SE} = \sqrt{(0.2496 + 0.2304)/150}, which is \displaystyle \sqrt{0.0032} \approx 0.0566. At 90\%: margin \approx 0.0931, interval 0.0269 to 0.2131, above 0, so evidence that coaching causes more households to lower the setting. At 99\%: margin \approx 0.1457, interval -0.0257 to 0.2657, which contains 0, so at that level the trial cannot rule out no effect.
- No. Only 6 households applied in the first group, fewer than 10, so the normal model is poor (the failures, 34 and 25, are fine).
- (a) Yes: random assignment balances everything but the letter. (b) No: owning or renting cannot be assigned, and the two groups may differ in income, type of home and much else. (c) No: households chose to ask, so interest in saving energy is mixed up with the scan.
- Every plausible value of p_A - p_B is negative, so B had the higher rate, by between 2 and 9 percentage points.
- For independent samples the variance of a difference is the sum of the two variances, \sigma^2_{\hat{p}_1} + \sigma^2_{\hat{p}_2}. The standard error is the square root of that sum. Adding the standard errors themselves would overstate the spread, because the two chance errors partly offset as often as they reinforce.
- The interval contains 0, but it also contains every value from -0.056 to 0.036. A gap of 3 points either way fits the data, as does no gap, so the data fail to show a difference without showing equality.
- The ten percent condition protects the independence of draws without replacement from a population; Maartje sampled from cities, Sjoerd sampled from nothing. In the experiment, the random assignment of letters is the source of chance that the interval measures.
- The 600 households would have to be a random sample of all Groningen households who have not applied (with the ten percent condition then checked). Randomly assigning the letters within that sample would let the causal conclusion extend to the whole group.
- Reversing the order turns an interval from a to b into one from -b to -a. If a \le 0 \le b, then -b \le 0 \le -a, and if 0 lies outside one interval it lies outside the other; only which side it is on changes.