Statistics · S.MD.B.7

Power and what raises it

A new dough might tear less often. If it really does, how likely is a taste-kitchen trial to notice, and how many dumplings must the kitchen boil to be fairly sure it will?

You need

Type I and Type II errorsStatistics

Where you are 7 in Hypothesis testing

Statistics7

Statistical inference8

Hypothesis testing7

▴ 2 above Test statistic Deciding and concluding from a p-value Type I and Type II errors Cautions about significance

Power and what raises it

Why this exists

A made-up pierogi kitchen in Krakow knows from long records that 15\% of its dumplings tear when boiled. Wojciech, the head cook, has a new dough. Malgorzata, who runs the kitchen, will switch only on evidence, so they plan a trial: boil a random sample of n dumplings made with the new dough and test

H_0\colon p = 0.15 \qquad H_a\colon p < 0.15

where p is the proportion of new-dough dumplings that tear, at \alpha = 0.05 (a One sample test for a proportion).

Type I and Type II errors already gave the two ways this test can go wrong, and showed that \beta, the chance of missing a false H_0, is computed at one specific alternative value. Before boiling anything, Malgorzata wants the question turned around: if the new dough really is better, what is the chance the trial says so? A trial that would usually miss a real improvement is a waste of flour. That chance is the power, and it is the number used to choose n.

Definition

The power of a test at a specific alternative value p_a is the probability that the test rejects H_0 when the true parameter value is p_a:

\begin{aligned} \text{power} &= P(\text{reject } H_0 \mid p = p_a) \\ &= 1 - \beta \end{aligned}

Rejecting and failing to reject are the only two decisions, so at the same p_a the power and the Type II error probability \beta add to 1. Like \beta, power has no single value: it is a number for each alternative value, and it also depends on n and \alpha.

Intuition

For a lower-tailed test of one proportion, “reject when the p-value is at most \alpha” is the same as “reject when \hat{p} falls below a cutoff c”. The cutoff is set in the world where H_0 is true, where \hat{p} is approximately normal with mean p_0 and standard deviation \displaystyle \sqrt{p_0(1 - p_0)/n} (see Sampling distribution of a sample proportion):

c = p_0 - z^*\sqrt{\frac{p_0(1 - p_0)}{n}}

with z^* = 1.645 for \alpha = 0.05. Power is then the area of a second curve, the one for the world where p = p_a, on the reject side of that same cutoff. In Figure 1 the dashed curve is the world where the new dough tears 10\% of the time; the shaded part of it, left of the cutoff, is the power.

Figure 1The trial with 200 dumplings (made-up data). Left of the cutoff the kitchen rejects the null hypothesis. The shaded area under the dashed curve is the chance of that when the true tear rate is 0.10, the power.

Anything that pulls the two curves apart, or makes them narrower, leaves more of the dashed curve past the cutoff. Figure 2 shows the main way to do that: a larger sample.

Results

Power at one alternative (lower-tailed test). With cutoff c from H_0,

\text{power} = P(\hat{p} < c \mid p = p_a)

computed with \hat{p} approximately normal with mean p_a and standard deviation \displaystyle \sqrt{p_a(1 - p_a)/n}. For an upper-tailed test it is P(\hat{p} > c \mid p = p_a); for a two-sided test it is the area beyond both cutoffs.

What raises power. With the other three held fixed, power is larger when

the sample size n is larger;
the significance level \alpha is larger, at the price of more Type I errors;
the true value p_a is farther from p_0 (a larger effect size);
the variability of the statistic is smaller.

Planning. Choose the alternative that matters in practice and a target power (often 0.80 or 0.90), then find the smallest n whose power at that alternative reaches the target.

A fail-to-reject from a low-power study is weak evidence. If the power at a value that matters was low, failing to reject is what the test would usually do even when that value is true.

Why it is true

Larger n. Both standard deviations shrink like \displaystyle 1/\sqrt{n}. The cutoff sits z^* null standard deviations below p_0, so it moves toward p_0 and away from p_a; meanwhile the alternative curve narrows around p_a. Each change puts more of the alternative curve below c.

Larger \alpha. A larger \alpha means a smaller z^*, so c moves toward p_0. The alternative curve does not move, so more of it lies on the reject side. The same move puts more of the null curve there too, which is the larger Type I chance.

Larger effect size. The cutoff depends only on p_0, n and \alpha. Moving p_a farther below it slides the whole alternative curve away from c, deeper into the reject side.

Less variability. The power is the area left of z = (c - p_a)/\sigma_1 under the standard normal curve, where \sigma_1 is the alternative standard deviation. For a fixed gap between c and p_a, a smaller \sigma_1 makes z larger. For a proportion, the spread is set by p and n: p(1 - p) is largest at p = 0.5, so the same drop in p is harder to detect near 0.5 than near 0 or 1.

Worked examples

Each z and power below is computed from the unrounded cutoff and standard deviations; redoing the arithmetic from the rounded values printed can change the last digit. All use the normal model, so they are approximate: the exact binomial power in Example 1 is about 0.648, close to the normal 0.655.

Example 1

Power of the 200-dumpling trial

Wojciech boils a random sample of 200 new-dough dumplings. Find the power of the test at p_a = 0.10, and say what it means.

Solution

The cutoff comes from H_0. The null standard deviation is

\sigma_0 = \sqrt{\frac{0.15 \times 0.85}{200}} \approx 0.02525

and the expected counts under H_0, 200(0.15) = 30 and 200(0.85) = 170, are both at least 10, so the normal model holds. For a lower tail at \alpha = 0.05,

c = 0.15 - 1.645(0.02525) \approx 0.1085

The kitchen rejects H_0 if fewer than 0.1085 of the 200 tear, that is, 21 or fewer.

Now move to the world where p = 0.10. The counts 200(0.10) = 20 and 180 are large enough, and

\sigma_1 = \sqrt{\frac{0.10 \times 0.90}{200}} \approx 0.02121

z = \frac{0.1085 - 0.10}{0.02121} \approx 0.40

\text{power} = P(Z < 0.40) \approx 0.655

In context: if the new dough really tears 10\% of the time, a 200-dumpling trial detects the improvement about two times in three, and misses it with probability \beta \approx 0.345. That is the shaded area in Figure 1.

Example 2

Doubling the sample

Find the power at p_a = 0.10 if the trial uses 400 dumplings instead.

Solution

Only n changes, but it changes both curves. Under H_0,

\sigma_0 = \sqrt{\frac{0.15 \times 0.85}{400}} \approx 0.01785

c = 0.15 - 1.645(0.01785) \approx 0.1206

The cutoff has moved from 0.1085 up toward 0.15. At p = 0.10,

\sigma_1 = \sqrt{\frac{0.10 \times 0.90}{400}} = 0.015

z = \frac{0.1206 - 0.10}{0.015} \approx 1.38

\text{power} = P(Z < 1.38) \approx 0.916

Figure 2Doubling the sample narrows both curves, so they overlap less. The cutoff moves toward 0.15 and nearly all of the dashed curve falls on the reject side.

Doubling the sample raises the power from about 0.66 to about 0.92, and cuts the chance of missing a 10\% dough from about one in three to about one in twelve. The Type I chance is still 0.05: a larger sample buys power without paying for it in false alarms.

Example 3

A larger significance level

Malgorzata cannot spare more than 200 dumplings. Jaroslaw, the kitchen’s buyer, suggests \alpha = 0.10 instead. Find the power at p_a = 0.10, and state the cost.

Solution

For a lower tail at \alpha = 0.10, z^* \approx 1.282, and \sigma_0 \approx 0.02525 as in Example 1:

c = 0.15 - 1.282(0.02525) \approx 0.1176

z = \frac{0.1176 - 0.10}{0.02121} \approx 0.83

\text{power} = P(Z < 0.83) \approx 0.797

The power rises from 0.655 to about 0.80. The cost: if the new dough is really no better (tear rate 0.15), the kitchen now wrongly switches 10\% of the time instead of 5\%. Whether that is worth it is a kitchen decision, not a mathematical one.

Seen as planning, \alpha and n trade against each other. At \alpha = 0.10 the target power 0.80 needs only 202 dumplings, while at \alpha = 0.05 it needs 283 (Example 6): the 81 dumplings saved are paid for with twice the rate of false alarms.

Example 4

Effect size: how much better the dough is

Keep n = 200 and \alpha = 0.05. Find the power if the new dough really tears 8\% of the time, and if it tears 12\% of the time.

Solution

The cutoff stays at 0.1085, since it depends only on H_0, n and \alpha.

At p_a = 0.08: \displaystyle \sigma_1 = \sqrt{0.08 \times 0.92/200} \approx 0.01918 (counts 16 and 184),

z = \frac{0.1085 - 0.08}{0.01918} \approx 1.48

so the power is about 0.931.

At p_a = 0.12: \displaystyle \sigma_1 = \sqrt{0.12 \times 0.88/200} \approx 0.02298,

z = \frac{0.1085 - 0.12}{0.02298} \approx -0.50

so the power is about 0.308.

The last column below is the smallest n with power at least 0.80 at that rate, found by the search of Example 6.

True tear rate p_a	Power, n = 200	n for power 0.80
0.08	0.931	136
0.10	0.655	283
0.12	0.308	824

A dough that cuts tearing almost in half is caught 93\% of the time; one that only trims it to 12\% is caught less than a third of the time. Small effects need large samples: shrinking the drop from 0.05 to 0.03 nearly triples the dumplings needed, so a plan must first name the smallest improvement worth finding.

Example 5

Variability: the same drop in a different place

Jaroslaw’s cousin runs a test where the null rate is 0.50: H_0\colon p = 0.50, H_a\colon p < 0.50, n = 200, \alpha = 0.05. Find the power at p_a = 0.45, and compare with Example 1, where the drop was also 0.05.

Solution

\sigma_0 = \sqrt{\frac{0.50 \times 0.50}{200}} \approx 0.03536

c = 0.50 - 1.645(0.03536) \approx 0.4418

\sigma_1 = \sqrt{\frac{0.45 \times 0.55}{200}} \approx 0.03518

z = \frac{0.4418 - 0.45}{0.03518} \approx -0.23

so the power is about 0.408.

The drop is 0.05 in both tests, yet the power is 0.408 here against 0.655 for the dough. The standard deviations near 0.5 (about 0.035) are larger than those near 0.1 to 0.15 (about 0.021 to 0.025), so the curves are wider and overlap more. Less variability means more power.

Example 6

Planning the sample size for power 0.80

Malgorzata wants a trial with power at least 0.80 at p_a = 0.10, with \alpha = 0.05. Build a table of power for several sample sizes, and find the smallest n that meets the target.

Solution

For each n the computation is the one in Example 1: cutoff from p_0 = 0.15, then the area left of it under the curve for p = 0.10. The counts under p = 0.10 need n \ge 100.

n	cutoff c	power
100	0.0913	0.385
200	0.1085	0.655
300	0.1161	0.824
400	0.1206	0.916
500	0.1237	0.962

The target lies between 200 and 300. Trying values between them:

n	power
250	0.751
280	0.797
282	0.7998
283	0.8012
290	0.811

The smallest sample with power at least 0.80 is n = 283 dumplings. Figure 3 draws the whole curve. It rises fast at first and then levels off: going from 100 to 200 dumplings gains about 0.27, going from 400 to 500 gains less than 0.05. This is the same kind of planning as choosing n for a Point estimates and the margin of error, except that the target is a chance of detection, not a width.

Figure 3Power at the true tear rate 0.10 for each sample size (made-up setting, alpha 0.05). The target 0.80 is first reached near 283 dumplings.

Example 7

What a small trial that fails to reject means

Before Malgorzata got involved, Wojciech had boiled 100 new-dough dumplings and 12 tore. He says, “The test was not significant, so the new dough is no better.” Carry out the test, find its power at p_a = 0.10, and judge his claim.

Solution

The counts under H_0 are 15 and 85, so the normal model applies. With \displaystyle \hat{p} = \frac{12}{100} = 0.12 and \displaystyle \sigma_0 = \sqrt{0.15 \times 0.85/100} \approx 0.03571,

z = \frac{0.12 - 0.15}{0.03571} \approx -0.84

and the p-value is P(Z \le -0.84) \approx 0.200.

Since 0.200 > 0.05, he fails to reject H_0. From the table in Example 6, the power of this trial at p_a = 0.10 was only 0.385: even if the new dough really tears 10\% of the time, a 100-dumpling trial fails to reject with probability about 0.61.

Failing to reject never shows H_0 is true, and with power this low it barely tells a better dough from a no-better one. The honest conclusion is “not enough evidence either way; run a larger trial”, which is exactly why Malgorzata plans one with power 0.80.

Boundaries

Power needs a specific alternative. “The power of the test” with no p_a named has no value. As p_a moves toward p_0, the power falls toward \alpha; it never falls below \alpha for an alternative on the side H_a names.

The model must hold in both worlds. The cutoff needs np_0 and n(1 - p_0) at least 10, and the power needs np_a and n(1 - p_a) at least 10, with a random sample and independent trials (see Conditions for one proportion). Otherwise the computed power is not the real one.

Power is computed before the data. It describes the plan. Plugging the observed \hat{p} in as p_a after the fact adds nothing the p-value did not already say.

Raising \alpha is not free, and a huge n can detect a difference too small to matter (see Cautions about significance). Planning starts from the smallest effect worth detecting.

Where this goes next

Every test in Hypothesis testing has a power, computed the same way: a cutoff from the null world, an area from an alternative world. For a mean, the spread is \displaystyle \sigma/\sqrt{n}, so reducing measurement noise becomes a second lever beside sample size. Power also explains the caution in Cautions about significance: a test with enormous power will reject for differences nobody cares about.

Exercises

Computing power

Exercises 1 and 2 use the dough test: H_0\colon p = 0.15, H_a\colon p < 0.15, \alpha = 0.05. Exercises 3 to 5 use another trial: Malgorzata tests a new menu, where p is the proportion of diners who order a second plate, with H_0\colon p = 0.30 and H_a\colon p > 0.30.

With n = 200, find the power if the new dough really tears 9\% of the time.
With n = 350, find the power at p_a = 0.10 and the value of \beta there. Say what each means.
With n = 150 diners and \alpha = 0.05, find the cutoff and the power at p_a = 0.40.
Repeat Exercise 3 with \alpha = 0.01.
Repeat Exercise 3 with n = 300 diners.
Jaroslaw tests whether the share of dumplings that come out underweight differs from 20\%: H_0\colon p = 0.20, H_a\colon p \neq 0.20, n = 400, \alpha = 0.05. Find both cutoffs and the power if the true share is 25\%.
Over a year the kitchen runs 40 separate 400-dumpling trials of doughs that each really tear 10\% of the time. About how many trials reject H_0?

What raises power, and planning

For the dough test with n = 200 and \alpha = 0.05, say whether the power at p_a = 0.10 rises, falls or stays the same: (a) n becomes 250; (b) \alpha becomes 0.01; (c) power is instead computed at p_a = 0.07; (d) power is instead computed at p_a = 0.13.
Trying sample sizes in steps of 50 from 300, find the smallest one with power at least 0.90 at p_a = 0.10 for the dough test.
Trying sample sizes in steps of 25 from 100, find the smallest one with power at least 0.90 at p_a = 0.40 for the menu test at \alpha = 0.05.
A rival kitchen tests H_0\colon p = 0.40 against H_a\colon p < 0.40 with n = 300 and \alpha = 0.05. Find the power at p_a = 0.35, and compare with the dough test at n = 300.
For the dough test, what is the smallest n for which the normal model can be used at p_a = 0.10? At p_a = 0.05?
Jaroslaw ran a 120-dumpling trial and failed to reject. Find its power at p_a = 0.10, and say what his result shows.
Wojciech says, “The power at 0.10 is 0.66, so there is a 66\% chance the new dough tears only 10\% of the time.” What is wrong with this?

Reasoning

Explain why power, like \beta, has no value until an alternative is named.
Explain why a larger sample raises power even though \alpha stays 0.05.
Explain why a kitchen should not raise power simply by choosing \alpha = 0.25.
Explain why failing to reject in a low-power study says little about H_0.
Explain why the power at p_a approaches \alpha as p_a moves toward p_0.
Planning n for a margin of error and planning n for power both pick a sample size before the data. What target does each aim at?
A chain of kitchens tests 40{,}000 dumplings and rejects H_0\colon p = 0.15 with \hat{p} = 0.147. Explain how power explains this, and why it may not matter.

Answers

Cutoff 0.1085; \displaystyle \sigma_1 = \sqrt{0.09 \times 0.91/200} \approx 0.02024, \displaystyle z = \frac{0.1085 - 0.09}{0.02024} \approx 0.91, power \approx 0.819.
\displaystyle \sigma_0 = \sqrt{0.15 \times 0.85/350} \approx 0.01909, c \approx 0.1186; \sigma_1 \approx 0.01604, z \approx 1.16, power \approx 0.877, \beta \approx 0.123. A 10\% dough is detected about 88\% of the time and missed about 12\% of the time.
\displaystyle \sigma_0 = \sqrt{0.30 \times 0.70/150} \approx 0.03742, c = 0.30 + 1.645(0.03742) \approx 0.3615; \displaystyle \sigma_1 = \sqrt{0.40 \times 0.60/150} = 0.04, \displaystyle z = \frac{0.3615 - 0.40}{0.04} \approx -0.96, power = P(Z > -0.96) \approx 0.832.
z^* \approx 2.326, c \approx 0.3870, z \approx -0.32, power \approx 0.627. Lower, as a smaller \alpha moves the cutoff away from p_a.
\sigma_0 \approx 0.02646, c \approx 0.3435; \sigma_1 \approx 0.02828, z \approx -2.00, power \approx 0.977.
\displaystyle \sigma_0 = \sqrt{0.20 \times 0.80/400} = 0.02, cutoffs 0.20 \pm 1.96(0.02), that is 0.1608 and 0.2392. At p = 0.25, \sigma_1 \approx 0.02165; z values \approx -4.12 and \approx -0.50, so power \approx 0.000 + 0.691 = 0.691.
Power 0.916 (Example 2), so about 40 \times 0.916 \approx 36.6, roughly 37 trials.
(a) Rises. (b) Falls. (c) Rises: 0.07 is farther from 0.15. (d) Falls: 0.13 is closer.
n = 300: 0.824; 350: 0.877; 400: 0.916. Smallest is 400.
n = 150: 0.832; 175: 0.877; 200: 0.911. Smallest is 200.
\displaystyle \sigma_0 = \sqrt{0.40 \times 0.60/300} \approx 0.02828, c \approx 0.3535; \sigma_1 \approx 0.02754, z \approx 0.13, power \approx 0.550, against 0.824 for the dough test at n = 300: the same drop of 0.05, more spread near 0.4.
n(0.10) \ge 10 needs n \ge 100 (then n(0.90) = 90; under H_0, 15 and 85). At p_a = 0.05, n(0.05) \ge 10 needs n \ge 200.
\sigma_0 \approx 0.03260, c \approx 0.0964; \sigma_1 \approx 0.02739, z \approx -0.13, power \approx 0.447. A 10\% dough would be missed more than half the time, so his result does not show the dough is no better.
Power is a chance about the trial’s outcome assuming p = 0.10, not a chance that p = 0.10.
Power is an area under the sampling distribution of \hat{p}, and where that curve sits depends on the true p. Each alternative gives its own curve and its own power.
With larger n both curves narrow; the cutoff stays z^* null standard deviations from p_0, which is now closer to p_0, so more of the alternative curve passes it while the null area beyond it is still \alpha.
Power would rise, but a quarter of truly no-better doughs would be adopted. A larger n raises power without that cost.
If the power was low, failing to reject was likely even if H_a is true, so the outcome hardly distinguishes the two.
As p_a \to p_0 the alternative curve slides onto the null curve, and its area beyond the cutoff becomes the null curve’s area there, which is \alpha.
The margin of error plan aims at an interval width; the power plan aims at a chance of rejecting H_0 at a chosen alternative.
With n = 40{,}000, \sigma_0 \approx 0.00179 and the cutoff is about 0.1471, only 0.003 below 0.15, so \hat{p} = 0.147 is rejected. The power is about 0.96 at a true rate of 0.144 and still about 0.51 at 0.147: the test reliably detects a drop of less than one percentage point, which is too small to matter to a kitchen.