Test for a difference of proportions
Two samples show different shares of some feature. Is the gap real, a sign that the two populations or the two treatments truly differ, or could chance alone have made it?
Where you are 5 in Inference for proportions
When you reach for it
Amani studies small businesses in Kenya. She wants to know whether the share of small shops that accept mobile money payments differs between two towns, Nakuru and Eldoret. She takes an independent random sample of shops in each: 132 of 200 Nakuru shops accept mobile money, and 117 of 220 Eldoret shops do (made-up data throughout this lesson). The sample shares are 0.66 and about 0.532. The gap is about 0.13, but two other samples would give another gap. Is this one too large to be chance, if the two towns really had the same share?
That is a yes-or-no question about p_1 - p_2, and the two-sample z test for a difference of proportions answers it. It joins two earlier lessons. The Interval for a difference of proportions estimated p_1 - p_2 as \hat{p}_1 - \hat{p}_2 plus or minus a margin, with a standard error built from both samples. The One sample test for a proportion measured how far a sample share sat from a claimed value, in standard deviations computed as if the null hypothesis were true. Here the claim is that the two shares are equal. Day 1 runs the test on two random samples; Day 2 runs it on a randomized experiment, adds one-sided alternatives, and sets the test beside the interval.
The procedure
Day 1: two independent random samples
Parameters and hypotheses. Name p_1 and p_2 in context: the proportions of two stated populations with a stated feature. The null hypothesis is no difference, and the alternative is what the investigator set out to find, chosen before the data are seen:
H_0 can be written p_1 = p_2. The alternative may instead be p_1 - p_2 > 0 or p_1 - p_2 < 0 (Day 2).
The pooled proportion. Every step of a test reasons inside H_0, as the p-values and the significance level lesson explained. If H_0 is true, the two populations share one proportion; call it p. The best estimate of that one share uses both samples together, all the successes over all the observations:
This is the pooled (combined) proportion. Since x_1 = n_1\hat{p}_1 and x_2 = n_2\hat{p}_2, it is also \displaystyle \frac{n_1\hat{p}_1 + n_2\hat{p}_2}{n_1 + n_2}, a weighted average of the two sample shares, weighted by sample size. It always lies between them, closer to the larger sample’s share.
The test statistic. The Sampling distribution of a difference lesson showed that for independent samples \hat{p}_1 - \hat{p}_2 has mean p_1 - p_2 and standard deviation \displaystyle \sqrt{p_1(1 - p_1)/n_1 + p_2(1 - p_2)/n_2}. Under H_0 both shares equal p, so the two terms share one factor, and \hat{p}_C stands in for the unknown p:
As in every Test statistic, z counts how many standard errors the estimate sits from the null value 0.
Why the test pools and the interval does not. An interval makes no claim about the shares; it estimates how far apart they are, so it must let them differ and uses \hat{p}_1 and \hat{p}_2 separately. A test assumes H_0 for the sake of argument, and H_0 says there is one share. Using two separate shares would compute the spread for a world the test is not assuming.
Conditions. They are the interval’s conditions, with one change in the counts:
- Random and independent. Each sample is random from its own population, and the two samples are drawn independently of each other.
- Ten percent. Each sample is at most a tenth of its own population.
- Large counts. n_1\hat{p}_C, n_1(1 - \hat{p}_C), n_2\hat{p}_C and n_2(1 - \hat{p}_C) are all at least 10. These are the counts each sample would be expected to show if H_0 were true, which is the world whose normal model the test uses.
P-value, decision, conclusion. The p-value is the chance, if H_0 were true, of a z at least as extreme as the one observed in the direction H_a names; for \neq both tails count. Compare it with \alpha and reject or fail to reject H_0, then say what that means about the two populations, naming \alpha and the p-value. A calculator’s 2-PropZTest takes x_1, n_1, x_2, n_2 and the alternative, and prints z, the p-value, \hat{p}_1, \hat{p}_2 and the pooled \hat{p}. It checks no condition.
Day 2: experiments, one-sided tests, and the interval
A randomized experiment. In a Completely randomized design and replication one group of subjects is split at random between two treatments, and p_1 and p_2 are the shares that would respond if all the subjects got treatment 1, or all got treatment 2. The random condition is Random assignment and confounding; nothing is sampled from a population, so the ten percent condition is not checked; the large counts use \hat{p}_C as before. Under H_0 the treatment makes no difference, so each subject would have responded the same either way, and the pooled share is the response rate of the whole group. A rejection now supports a claim of cause: random assignment leaves the groups alike, apart from chance, in everything but the treatment (Observational studies and experiments). The claim covers the subjects in the experiment, and reaches a wider population only if they were sampled at random from it.
One-sided alternatives. If the question, fixed in advance, is whether treatment 1 does better, then H_a\colon p_1 - p_2 > 0, and the p-value is the upper tail alone. A result in the other direction cannot support it however large it is.
Test and interval together. A two-sided test at level \alpha and a (1 - \alpha) confidence interval answer nearly the same question, so they nearly always agree: reject H_0 when the interval leaves out 0. A one-sided test at \alpha = 0.05 matches the side of a 90\% interval. The match is close but not exact, because the test’s standard error is pooled and the interval’s is not. When a result sits right on the line, the two can land on opposite sides.
Worked examples
Day 1: mobile money in two towns
Carry out Amani’s test at \alpha = 0.05: 132 of 200 Nakuru shops and 117 of 220 Eldoret shops accept mobile money. Take each town, as part of the problem, to have more than 3{,}000 small shops.
Parameters. Let p_N be the proportion of all small shops in Nakuru that accept mobile money and p_E the proportion in Eldoret. She had no reason in advance to expect one town higher, so the test is two-sided:
Pooled proportion. If H_0 is true there is one share, estimated from all 420 shops:
Conditions. Two random samples, drawn separately in each town, so random and independent. A tenth of 3{,}000 is 300, more than both 200 and 220. Expected counts with the pooled share: 200 \times 0.5929 \approx 118.6 and 200 \times 0.4071 \approx 81.4 in Nakuru, 220 \times 0.5929 \approx 130.4 and 220 \times 0.4071 \approx 89.6 in Eldoret, all at least 10.
Test statistic. \hat{p}_N = 132/200 = 0.66 and \hat{p}_E = 117/220 \approx 0.5318.
P-value. Both tails count, as Figure 1 shows: 2P(Z \ge 2.67) \approx 2(0.0038) \approx 0.0076.
Decision and conclusion. 0.0076 \le 0.05, so Amani rejects H_0. If the two towns had equal shares, samples this size would show a gap of 0.128 or more, either way, less than 1\% of the time. There is convincing evidence that the share of small shops accepting mobile money differs between Nakuru and Eldoret, and the samples point to Nakuru’s being higher. These are two populations compared, not an experiment, so the test says nothing about why.
Day 1: Sunday opening, with 2-PropZTest output
The same shops record whether they open on Sunday: 86 of the 200 Nakuru shops and 88 of the 220 Eldoret shops do. Amani enters x_1 = 86, n_1 = 200, x_2 = 88, n_2 = 220 and the alternative p_1 \neq p_2.
| Screen line | What it is |
|---|---|
| p_1 \neq p_2 | the alternative |
| z = .6233 | the test statistic |
| p = .5331 | the p-value |
| \hat{p}_1 = .43, \hat{p}_2 = .4 | the two sample shares |
| \hat{p} = .4143 | the pooled share \hat{p}_C |
Confirm \hat{p}_C and z by hand and conclude at \alpha = 0.05.
SolutionThe pooled share is \displaystyle \frac{86 + 88}{420} = \frac{174}{420} \approx 0.4143, matching the last line. The calculator does not check conditions: the samples are the same random, independent ones, and the expected counts 200 \times 0.4143 \approx 82.9, 117.1, 220 \times 0.4143 \approx 91.1 and 128.9 are all at least 10.
which agrees with the screen. Since 0.533 > 0.05, Amani fails to reject H_0. A gap of 3 percentage points is the kind chance produces about half the time when the shares are equal, so the data give no convincing evidence that the share of shops open on Sunday differs between the towns. They do not show the shares are equal: a small real gap would also produce samples like these, and missing it would be a Type I and Type II errors.
Day 1: conditions checked with the pooled share
Kioko takes smaller random samples, 60 shops in Nakuru and 80 in Eldoret, and asks whether the shop takes bank cards: 9 and 21 do. Test at \alpha = 0.05 whether the shares differ.
SolutionHypotheses. p_N and p_E are the proportions of all small shops in each town that take bank cards; H_0\colon p_N - p_E = 0, H_a\colon p_N - p_E \neq 0.
Conditions. Random, independent samples, each far below a tenth of more than 3{,}000 shops. The pooled share is \displaystyle \hat{p}_C = \frac{30}{140} \approx 0.2143, and the expected counts are 60 \times 0.2143 \approx 12.9, 60 \times 0.7857 \approx 47.1, 80 \times 0.2143 \approx 17.1 and 80 \times 0.7857 \approx 62.9, all at least 10. The test may go ahead. Note that the Nakuru sample shows only 9 card shops: an interval, which checks the observed counts, could not be built from these samples, but the test checks the counts expected under H_0, and those pass.
Statistic. \hat{p}_N = 9/60 = 0.15 and \hat{p}_E = 21/80 = 0.2625.
P-value. 2P(Z \le -1.61) \approx 0.108.
Conclusion. 0.108 > 0.05, so Kioko fails to reject H_0: these samples give no convincing evidence that the share of shops taking cards differs between the towns. With 60 and 80 shops even a gap of 11 points is within what chance does fairly often; Amani’s larger samples would detect a gap that size far more reliably.
Day 2: two text messages, a one-sided test
Rehema works for a mobile-money company. From the member list of Kioko’s shop owners’ association in Eldoret she takes 500 owners whose shops do not yet accept mobile money, and assigns 250 at random to text A, a plain offer of a merchant account, and 250 to text B, the same offer plus a line saying how many nearby shops already accept mobile money. Before sending, the company states that it expects text B to do better. Within 30 days 68 of the text B owners and 47 of the text A owners sign up. Test at \alpha = 0.05.
SolutionParameters. p_B and p_A are the shares of these 500 owners who would sign up if all received text B, or all received text A. The question fixed in advance points one way:
Conditions. The texts were assigned at random. Nothing was sampled from a population, so there is no ten percent check. Pooled share \displaystyle \hat{p}_C = \frac{68 + 47}{500} = 0.23; expected counts 250 \times 0.23 = 57.5 and 250 \times 0.77 = 192.5 in each group, all at least 10.
Statistic. \hat{p}_B = 68/250 = 0.272, \hat{p}_A = 47/250 = 0.188.
P-value. Upper tail only: P(Z \ge 2.23) \approx 0.0128.
Conclusion. 0.0128 \le 0.05, so Rehema rejects H_0. The texts were randomly assigned, so this is convincing evidence that text B causes a higher sign-up rate than text A among these 500 owners. They came from one association’s list in one town, not a random sample of Kenyan shop owners, so the result reaches other owners only as far as they resemble these.
Day 2: the same experiment as an interval
Construct 95\% and 90\% intervals for p_B - p_A from Rehema’s data, and compare them with a two-sided test at \alpha = 0.05 and with her one-sided test.
SolutionThe interval lets the shares differ, so its standard error uses each group’s own share:
so the intervals are 0.084 \pm 0.0734 and 0.084 \pm 0.0616.
The 95\% interval runs from 0.0106 to 0.1574 and the 90\% interval from 0.0224 to 0.1456. A two-sided test would have a p-value of 2(0.0128) \approx 0.0256 and reject at 0.05; the 95\% interval leaves out 0. The one-sided test rejected at 0.05; the 90\% interval, whose lower end cuts off 5\% in one tail, lies wholly above 0. Both pairs agree, and the interval adds what the test cannot: text B raises sign-up by somewhere between about 1 and 16 percentage points.
Day 2: when the test and the interval part ways
Kioko runs a follow-up with 400 other members whose shops do not take mobile money. Agent visits cost more than texts, so 100 owners are assigned at random to an in-person visit and 300 to text B. He fixes a two-sided test at \alpha = 0.05, since a visit could put some owners off. Signed up: 68 of 100 visited, 171 of 300 texted. Run the test, build the 95\% interval, and explain the result.
Test. \hat{p}_V = 0.68, \hat{p}_T = 171/300 = 0.57, and \displaystyle \hat{p}_C = \frac{239}{400} = 0.5975. Expected counts 59.75, 40.25, 179.25 and 120.75 are all at least 10, and the visits were randomly assigned.
The p-value is 2P(Z \ge 1.94) \approx 0.052 > 0.05: fail to reject.
Interval.
so the interval is 0.11 \pm 0.1072, from 0.0028 to 0.2172, which leaves out 0 (Figure 2).
Why they differ. The \displaystyle \frac{1}{100} term dominates both standard errors, so the visit group’s spread matters most. The test gives that group the pooled spread 0.5975 \times 0.4025 \approx 0.2405; the interval gives it its own, 0.68 \times 0.32 = 0.2176, which is smaller. The pooled standard error is larger, so z is smaller. Neither calculation is wrong: they assume different things. Kioko planned a test, so he reports it: no convincing evidence at \alpha = 0.05, with a p-value of 0.052 and an interval that nearly touches 0. Both say the same honest thing, that the evidence for visits is right at the edge, and a larger visit group would settle it.
Boundaries
- The two samples must be independent. Paired data, such as the same shops asked twice, make \hat{p}_1 and \hat{p}_2 move together and the standard error wrong.
- Each group needs randomness: separate random samples, or random assignment. Volunteers or self-chosen groups give a p-value with nothing behind it.
- The large counts are checked with \hat{p}_C, at least 10 expected successes and failures in each group. Below that the normal model can misstate the p-value.
- The pooled standard error is right only when the null value is 0. A null hypothesis such as p_1 - p_2 = 0.05 does not say the shares are equal, so pooling has no basis.
- Rejecting H_0 shows a difference; only random assignment shows a cause, and then only for the subjects in the experiment unless they were randomly sampled.
- Failing to reject is not evidence that the shares are equal.
- The p-value covers chance variation only, not nonresponse or a slanted question.
Where this goes next
A difference of shares compares two groups on a yes-or-no variable. With three or more groups, or a variable with more than two categories, the Tests for independence and homogeneity takes over; for two groups and two categories its statistic is the square of this z. For averages rather than shares, the Test for a difference of means follows the same plan without pooling.
Exercises
Use z^* = 1.645 and 1.96 for 90\% and 95\% confidence. All data are made up. Unless told otherwise, town samples are independent random samples, and each town has more than 3{,}000 small shops.
Day 1: two independent random samples
- 150 of 250 Nakuru shops and 126 of 240 Eldoret shops sell phone airtime. Test at \alpha = 0.05 whether the shares differ, with conditions and a conclusion in context.
- A sample of 160 Nakuru shops has \hat{p}_1 = 0.35 selling bottled water by the crate, and a sample of 240 Eldoret shops has \hat{p}_2 = 0.45. Find x_1, x_2 and \hat{p}_C, and say why \hat{p}_C is nearer 0.45 than 0.35.
- For each plan, say whether the two-sample z test may be used, and if not, which condition fails. (a) Random samples of 150 and 180 shops, with 4 and 9 selling solar lamps. (b) A random sample of 120 of the 800 stalls in one Nakuru market, against a random sample of 200 Eldoret shops. (c) Shops whose owners phoned in to a radio program in each town. (d) Random samples of 200 and 250 shops, with 30 and 45 successes.
- A 2-PropZTest screen for p_1 \neq p_2 reads z = 2.3828, p = .0172, \hat{p}_1 = .4, \hat{p}_2 = .2778, \hat{p} = .3353, n_1 = 160, n_2 = 180. Find x_1 and x_2, confirm the pooled share, and give the decision at \alpha = 0.05 and at \alpha = 0.01.
- 90 of 300 Nakuru shops and 110 of 300 Eldoret shops offer home delivery. Test at \alpha = 0.10 whether the shares differ.
- For Amani’s mobile-money samples, compute the unpooled standard error and the z it would give. Compare with z \approx 2.67.
Day 2: experiments, one-sided tests, and the interval
- Of 250 member shop owners, 120 are assigned at random to a training session on a bookkeeping app and 130 to a leaflet about it. After two months 58 of the trained owners and 41 of the leaflet owners use the app. Test at \alpha = 0.05 whether training raises the share using the app.
- Write H_0 and H_a for each, and say whether the alternative is one- or two-sided. (a) Rehema suspects a lower monthly fee will raise sign-up, compared with the standard fee. (b) Kioko wonders whether a message in Swahili and one in English get different reply rates. (c) Amani asks whether the share of shops with a back-up generator differs between the two towns.
- Rehema fixed H_a\colon p_1 - p_2 > 0 before an experiment, but the calculator was left on p_1 \neq p_2 and printed z = 1.80, p = .0719. Find the correct p-value and decide at \alpha = 0.05. What would it have been had the screen shown z = -1.80?
- Of 300 owners, 150 are assigned at random to a reminder phone call and 150 to no call; 63 and 45 sign up. Carry out a two-sided test at \alpha = 0.05 and build a 95\% interval for p_{\text{call}} - p_{\text{none}}. Do they agree?
- A trial assigns 30 owners at random to each of two offers, and 3 and 6 sign up. May the two-sample z test be used?
- Kioko’s association members chose whether to join the association, and Rehema’s texts were assigned at random among them. Her test rejected H_0. To whom does the conclusion that text B causes more sign-ups apply, and what would be needed to extend it?
Reasoning
- Explain why the test pools the two samples to estimate one share while the interval for the same data does not.
- In Kioko’s card example the Nakuru sample had only 9 successes, yet the test’s conditions held. Explain why the test checks expected rather than observed counts.
- Amani’s Sunday test failed to reject. Explain why she may not report that the two towns have the same share of shops open on Sunday.
- Show that \displaystyle \hat{p}_C = \frac{n_1\hat{p}_1 + n_2\hat{p}_2}{n_1 + n_2} always lies between \hat{p}_1 and \hat{p}_2.
- Explain what happens to z, to a two-sided p-value, and to a one-sided alternative when the order of subtraction is reversed.
- Amani’s test showed that Nakuru’s share is higher. Explain why it cannot show that anything about Nakuru causes more shops to accept mobile money.
Answers
- p_N, p_E: shares of all small shops in each town selling airtime; H_a\colon p_N - p_E \neq 0. \hat{p}_C = 276/490 \approx 0.5633; expected counts 140.8, 109.2, 135.2, 104.8; 250 and 240 are under 300. \hat{p}_N = 0.60, \hat{p}_E = 0.525, \text{SE} \approx 0.0448, z \approx 1.67, p-value \approx 0.094 > 0.05. Fail to reject: no convincing evidence that the airtime shares differ between the towns.
- x_1 = 0.35 \times 160 = 56, x_2 = 0.45 \times 240 = 108, \hat{p}_C = 164/400 = 0.41. It is a weighted average, 0.4(0.35) + 0.6(0.45) = 0.41, and the Eldoret sample carries the larger weight, 240/400 = 0.6.
- (a) No: \hat{p}_C = 13/330 \approx 0.0394, and 150 \times 0.0394 \approx 5.9 is below 10. (b) No: 120 is more than a tenth of 800, which is 80. (c) No: the shops chose themselves. (d) Yes: \hat{p}_C = 75/450 \approx 0.1667, expected counts 33.3, 166.7, 41.7, 208.3.
- x_1 = 0.4 \times 160 = 64, x_2 \approx 0.2778 \times 180 \approx 50, and 114/340 \approx 0.3353. At \alpha = 0.05: 0.0172 \le 0.05, reject. At \alpha = 0.01: 0.0172 > 0.01, fail to reject. The same data are convincing at one level and not the other, which is why \alpha is fixed in advance.
- \hat{p}_C = 200/600 \approx 0.3333, expected counts 100 and 200 in each town. \hat{p}_N = 0.30, \hat{p}_E \approx 0.3667, \text{SE}^2 = 0.3333 \times 0.6667 times 2/300, so \text{SE} \approx 0.0385, z \approx -1.73, p-value \approx 0.083 \le 0.10. Reject: at the 0.10 level there is convincing evidence the delivery shares differ, with Eldoret’s higher.
- \text{SE}^2 = 0.66 \times 0.34/200 plus 0.5318 \times 0.4682/220, so \text{SE} \approx 0.0475 and z \approx 0.1282/0.0475 \approx 2.70, against 2.67 pooled. The pooled share 0.5929 has \hat{p}_C(1 - \hat{p}_C) \approx 0.2414, between Nakuru’s 0.2244 and Eldoret’s 0.2490, so the two standard errors are close and the decision is the same.
- H_a\colon p_T - p_L > 0. Random assignment; \hat{p}_C = 99/250 = 0.396, expected counts 47.5, 72.5, 51.5, 78.5. \hat{p}_T \approx 0.4833, \hat{p}_L \approx 0.3154, \text{SE} \approx 0.0619, z \approx 2.71, p-value \approx 0.0033 \le 0.05. Reject: convincing evidence that training causes a higher share of these owners to use the app.
- (a) H_0\colon p_{\text{low}} - p_{\text{std}} = 0, H_a\colon p_{\text{low}} - p_{\text{std}} > 0, one-sided. (b) H_0\colon p_S - p_E = 0, H_a\colon p_S - p_E \neq 0, two-sided. (c) H_0\colon p_N - p_E = 0, H_a\colon p_N - p_E \neq 0, two-sided.
- The data lie in the direction of H_a, so the p-value is one tail, 0.0719/2 \approx 0.036 \le 0.05: reject. With z = -1.80 the p-value would be P(Z \ge -1.80) \approx 0.964: no evidence at all for p_1 > p_2.
- \hat{p}_C = 108/300 = 0.36, expected counts 54 and 96 per group. \hat{p}_{\text{call}} = 0.42, \hat{p}_{\text{none}} = 0.30, test \text{SE} \approx 0.0554, z \approx 2.17, p-value \approx 0.030 \le 0.05: reject. Interval \text{SE} \approx 0.0550, 0.12 \pm 0.1078, from 0.0122 to 0.2278, which leaves out 0. They agree: the call raises sign-up, by about 1 to 23 points for these owners.
- No. \hat{p}_C = 9/60 = 0.15, and 30 \times 0.15 = 4.5 expected successes per group is below 10.
- To the 500 owners in the experiment. Random assignment gives cause within that group; to extend it to, say, all Eldoret shop owners without mobile money, the 500 would have to be a random sample of those owners.
- The interval estimates how far apart the shares are, so it cannot assume they are equal and uses each sample’s share. The test computes how \hat{p}_1 - \hat{p}_2 would vary if H_0 were true; under H_0 there is one share, and pooling both samples gives the best estimate of it.
- The test’s normal model is the one that holds if H_0 is true, so the counts that matter are those expected in that world, n_1\hat{p}_C and the rest. The interval’s model is built from each sample’s own share, so it checks the observed counts.
- Failing to reject means the data fit equal shares, but they fit a small real gap just as well; the test could not tell those apart. Claiming equality would risk a Type I and Type II errors presented as a finding.
- Say \hat{p}_1 \le \hat{p}_2. Then n_2\hat{p}_2 \ge n_2\hat{p}_1, so the numerator n_1\hat{p}_1 + n_2\hat{p}_2 is at least (n_1 + n_2)\hat{p}_1, and \hat{p}_C \ge \hat{p}_1; replacing \hat{p}_1 by \hat{p}_2 the same way gives \hat{p}_C \le \hat{p}_2.
- \hat{p}_C and the standard error do not change, and the estimate changes sign, so z becomes -z. A two-sided p-value uses |z| and is unchanged. A one-sided alternative must flip with it: p_1 - p_2 > 0 becomes p_2 - p_1 < 0, the same claim.
- The towns were sampled, not assigned. They differ in many ways at once (size, banks, phone coverage, kinds of shop), and any of these could be behind the gap. Only randomly assigning a treatment separates its effect from everything else.