Statistics

Cautions about significance

A test on 400,000 website visitors says a new checkout button “significantly” raises sales. Does that mean the gain is big, or worth paying for, and what changes when twenty such tests are run at once?

You need

Deciding and concluding from a p-valueStatistics

Where you are 7 in Hypothesis testing

Statistics7

Statistical inference8

Hypothesis testing7

▴ 2 above Test statistic Deciding and concluding from a p-value Type I and Type II errors

Cautions about significance

Power and what raises it

Why this exists

Leticia sells acai bowls through a made-up online shop in Brazil. Her web developer Vinicius tries changes on the site, and her analyst Murilo tests them: each visitor is shown one version, chosen by Random assignment and confounding, so any real difference in buying can be put down to the version. All data in this lesson are made up.

The Deciding and concluding from a p-value lesson ended every test the same way: compare the p-value with \alpha, then say whether there is convincing evidence for the alternative. That decision answers exactly one question: could chance alone plausibly explain the data? It does not say how big the effect is, whether it is worth acting on, or how the answer changes when many tests are run. Those three gaps are the subject here.

Definition

A result is statistically significant at level \alpha when its p-value is at or below \alpha: if the null hypothesis were true, data at least this far from it would turn up with probability at most \alpha. It is a statement about chance, not about size or importance.

A difference is practically significant when it is large enough to matter for the decision at hand: more sales than the change costs, a health benefit worth a side effect. It is judged from the size of the effect, estimated with a confidence interval, against a standard that comes from the context, never from the p-value.

Multiple comparisons means running many tests on one study, for example comparing several versions each against the original, or one comparison on many outcomes. Each test at level \alpha has chance \alpha of rejecting a true null hypothesis, so across many tests some false rejections become likely.

Intuition

The test statistic counts standard errors, and the standard error shrinks as the sample grows. A tiny difference measured on a huge sample is many standard errors from 0; a large difference measured on a small sample can be fewer than two. So significance mixes two things: how big the effect is and how much data there is.

Figure 1Two experiments at the acai shop (made-up data). The huge one rejects “no difference” but pins the gain near 0.004; the small one fails to reject, yet leaves gains as large as 0.12 plausible.

In Figure 1 the checkout-button test (Example 1, 200{,}000 visitors per version) has an interval so narrow that it clears 0, yet every value in it is under one percentage point. The photo-menu test (Example 4, 160 per version) has an interval that crosses 0, yet reaches gains of twelve points. The first is significant and small; the second is not significant and possibly large. The intervals show both facts; the p-values show neither.

Many tests work like many lottery tickets: one test at \alpha = 0.05 rarely raises a false alarm, but twenty give twenty chances (Figure 3, Example 5).

Results

Significance grows with the sample. For two groups of n each with a true difference d \ne 0 in proportions near a common p, the test statistic is about

z \approx \frac{d\sqrt{n}}{\sqrt{2p(1-p)}}

so z grows like \displaystyle \sqrt{n}. Quadrupling the sample doubles z, and any nonzero difference becomes significant once n is large enough.

Many tests. Run k tests, each at level \alpha, when every null hypothesis is true. Then

\text{expected number of false rejections} = k\alpha

and, when the tests are independent,

P(\text{at least one false rejection}) = 1 - (1 - \alpha)^k

A stricter level. Testing each of k hypotheses at level \alpha / k (the Bonferroni approach) keeps the chance of at least one false rejection at or below \alpha, whether or not the tests are independent.

Why it is true

Growth with n. The pooled standard error \displaystyle \sqrt{\hat{p}_C(1 - \hat{p}_C)(2/n)} has \displaystyle \sqrt{n} in its denominator. As n grows the observed difference settles near d while the standard error falls toward 0, so z grows without bound.

Expected count. Let X_i be 1 if test i falsely rejects and 0 if not. Under a true null hypothesis, P(X_i = 1) = \alpha, so each X_i has mean \alpha. The number of false rejections is X_1 + \cdots + X_k, and the mean of a sum is the sum of the means, k\alpha. That step needs no independence.

At least one. “At least one false rejection” is the complement of “none”. Each test avoids a false rejection with probability 1 - \alpha, and for independent tests the chances multiply, so P(\text{none}) = (1 - \alpha)^k.

The stricter level. The chance that at least one of several events happens is at most the sum of their chances, since adding counts any overlap more than once. With each test at \alpha / k, that sum is k \cdot \alpha/k = \alpha.

Worked examples

Example 1

A significant checkout button

Vinicius built version B of the checkout button. Over a month, 200{,}000 visitors saw version B and 22{,}400 of them bought; 200{,}000 saw the original, version A, and 21{,}600 bought. Test whether B changes the purchase rate at \alpha = 0.05 and give a 95\% interval for the difference.

Solution

Let p_B and p_A be the true purchase rates for visitors shown each version. Then H_0\colon p_B - p_A = 0 and H_a\colon p_B - p_A \ne 0. The versions were randomly assigned and every count is in the thousands, so the conditions of the two-sample z test hold. The sample rates are

\hat{p}_B = \frac{22{,}400}{200{,}000} = 0.112 \qquad \hat{p}_A = \frac{21{,}600}{200{,}000} = 0.108

The test assumes one shared rate, so it pools: \displaystyle \hat{p}_C = \frac{44{,}000}{400{,}000} = 0.11, and

\text{SE} = \sqrt{0.11(0.89)\left(\frac{2}{200{,}000}\right)} \approx 0.000989

z = \frac{0.112 - 0.108}{0.000989} \approx 4.04

The p-value is 2\,P(Z \ge 4.04) \approx 0.00005, far below 0.05: reject H_0. There is convincing evidence that version B changes the purchase rate.

The interval does not pool:

\text{SE} = \sqrt{\frac{0.112(0.888)}{200{,}000} + \frac{0.108(0.892)}{200{,}000}} \approx 0.000989

0.004 \pm 1.96(0.000989) \approx 0.004 \pm 0.0019

so the difference is about 0.0021 to 0.0059. Murilo can be 95\% confident that version B raises the purchase rate by between 0.2 and 0.6 of a percentage point, that is, between about 2 and 6 extra purchases per 1{,}000 visitors. The evidence is overwhelming; the effect is small.

Example 2

Is the gain worth it?

Version B works only with a payment service that charges 3{,}000 dollars a month. The shop has about 150{,}000 visitors a month and makes about 4 dollars of profit per order. Does the interval from Example 1 show that switching pays?

Solution

The switch pays when the extra profit covers 3{,}000 dollars. A gain of d in the purchase rate brings 150{,}000\,d extra orders, worth 4 \times 150{,}000\,d = 600{,}000\,d dollars, so the break-even gain is

d = \frac{3{,}000}{600{,}000} = 0.005

At the ends of the interval the extra profit is 600{,}000(0.0021) = 1{,}260 dollars and 600{,}000(0.0059) = 3{,}540 dollars a month (Figure 2).

Figure 2The checkout-button interval against two lines (made-up data). It clears 0, so the gain is statistically significant, but most of it sits below the 0.005 the change needs to pay for itself.

Most of the interval lies below break-even. So the data show that B is better than A, but not that B is better by enough to pay for itself. The test could never have said this: the standard 0.005 came from the shop’s costs, not from the data.

Example 3

The same gap in a smaller study

Suppose the button test had used 2{,}000 visitors per version, with the same rates: 224 and 216 purchases. Find z and the p-value. Then find roughly how many visitors per version make a gap of 0.004 significant at \alpha = 0.05, two-sided.

Solution

The pooled rate is again \displaystyle \hat{p}_C = \frac{440}{4{,}000} = 0.11, but now

\text{SE} = \sqrt{0.11(0.89)\left(\frac{2}{2{,}000}\right)} \approx 0.00989

z = \frac{0.004}{0.00989} \approx 0.40

The p-value is 2\,P(Z \ge 0.40) \approx 0.69: fail to reject. The gap is the same; only the standard error changed, by a factor of \displaystyle \sqrt{100} = 10, and z changed with it.

A two-sided test at 0.05 rejects once z \ge 1.96. Setting the Results formula equal to 1.96,

\begin{aligned} &\frac{0.004\sqrt{n}}{\sqrt{2(0.11)(0.89)}} = 1.96 \\ &n = 0.1958\left(\frac{1.96}{0.004}\right)^2 \approx 47{,}000 \end{aligned}

A gap ten times smaller would need a hundred times as many visitors, but some sample size always gets there.

Example 4

A small study that misses

Vinicius tried a menu with photos of each bowl on 160 visitors per version: 24 bought with the photo menu and 16 with the plain one. Test at \alpha = 0.05, give a 95\% interval, and say what Leticia should conclude.

Solution

\displaystyle \hat{p}_P = \frac{24}{160} = 0.15, \displaystyle \hat{p}_O = \frac{16}{160} = 0.10, and every count (24, 136, 16, 144) is at least 10. Pooling, \displaystyle \hat{p}_C = \frac{40}{320} = 0.125:

\text{SE} = \sqrt{0.125(0.875)\left(\frac{2}{160}\right)} \approx 0.0370

z = \frac{0.15 - 0.10}{0.0370} \approx 1.35

with p-value 2\,P(Z \ge 1.35) \approx 0.18. Fail to reject. For the interval,

\text{SE} = \sqrt{\frac{0.15(0.85)}{160} + \frac{0.10(0.90)}{160}} \approx 0.0369

0.05 \pm 1.96(0.0369) \approx 0.05 \pm 0.072

about -0.022 to 0.122 (Figure 1). There is not convincing evidence that the photos change the purchase rate, but the interval runs from a small loss to a gain of 12 points, which would be huge for the shop. The study simply had too little data: its power to detect even a large gain was low, so a Type II error is quite possible. The honest conclusion is “not yet known”, and the next step is a larger test, not dropping the photos.

Example 5

Twenty button colors

Vinicius then tests 20 new button colors, each against the original color on its own fresh group of visitors, each at \alpha = 0.05. Suppose that no color truly changes the purchase rate. Find the expected number of false rejections and the chance of at least one.

Solution

Every null hypothesis is true, so each test rejects with probability 0.05. The expected number of rejections, all of them false, is

k\alpha = 20(0.05) = 1

The groups are separate, so treat the tests as independent:

P(\text{at least one}) = 1 - 0.95^{20} \approx 1 - 0.358 = 0.642

About two times in three, at least one color would look “significant” by chance alone (Figure 3), so one winner among twenty is what chance produces.

Figure 3The chance that at least one of k independent tests at \alpha = 0.05 rejects a true null hypothesis, 1 - 0.95^k.

Example 6

Holding the twenty tests honest

Of the 20 colors, lime green gave the only p-value below 0.05: 0.012. Murilo uses the stricter level \alpha / k to keep the overall chance of any false rejection near 0.05. Decide for lime green, find the overall chance under that level, and judge a fresh confirming test: 3{,}282 of 30{,}000 new visitors bought with lime green and 3{,}240 of 30{,}000 with the original.

Solution

The stricter level is \displaystyle \frac{0.05}{20} = 0.0025. Since 0.012 > 0.0025, lime green does not pass. With every test at 0.0025, the chance of at least one false rejection among 20 independent tests is

1 - 0.9975^{20} \approx 0.049

just under 0.05, as the bound promised. The cost is power.

The fresh test settles it with data that played no part in picking lime green. The rates are \displaystyle \frac{3{,}282}{30{,}000} = 0.1094 and \displaystyle \frac{3{,}240}{30{,}000} = 0.108, the pooled rate is \displaystyle \frac{6{,}522}{60{,}000} = 0.1087, and

\text{SE} = \sqrt{0.1087(0.8913)\left(\frac{2}{30{,}000}\right)} \approx 0.00254

z = \frac{0.1094 - 0.108}{0.00254} \approx 0.55

with p-value 2\,P(Z \ge 0.55) \approx 0.58. Not confirmed. The honest report states that 20 colors were tested, that one reached 0.012, and that it did not hold up on new visitors.

Example 7

Forty changes, three winners

Over a year Murilo ran 40 independent tests at \alpha = 0.05, and 3 rejected. If none of the 40 changes truly mattered, how many rejections would be expected, and how likely are 3 or more?

Solution

The expected count is 40(0.05) = 2. The number of rejections, when every null is true, is a count of successes in 40 independent trials with chance 0.05 each, so it follows a The binomial distribution with n = 40 and p = 0.05:

\begin{aligned} P(0) &= 0.95^{40} \approx 0.1285 \\ P(1) &= 40(0.05)(0.95^{39}) \approx 0.2706 \\ P(2) &= \tbinom{40}{2}(0.05^2)(0.95^{38}) \approx 0.2777 \end{aligned}

so P(3 \text{ or more}) \approx 1 - 0.6767 = 0.323. Three winners from forty would happen about one year in three even if nothing worked. The three results are leads to retest, not findings.

Boundaries

Significance is tied to one test planned in advance. The level \alpha is the false-alarm rate of a single test whose hypotheses were fixed before the data. Picking the best of many results afterward, or choosing what to test after looking, makes the real false-alarm rate larger than \alpha by an unknown amount.

The formula 1 - (1 - \alpha)^k needs independence. Tests on the same visitors, or on related outcomes, are not independent; then the exact chance differs, though k\alpha still gives the expected count and \alpha / k still keeps the chance at most \alpha.

\alpha / k is one approach, not the rule. It is safe but strict, and with many tests it lowers power sharply. Naming the main test in advance and confirming with new data are other honest responses.

Practical importance comes from outside the data. How large a gain is worth having is set by the context (a cost, a risk, a benefit), ideally before the study.

Where this goes next

Every test that follows, for means in One sample test for a mean and Test for a difference of means, and for slopes, carries these cautions: report an interval beside the p-value, and say how many tests were run. Planning a study so that an effect worth caring about is likely to be detected is the work of power, and the two ways a decision can go wrong are set out in Type I and Type II errors.

Exercises

Statistical and practical significance

Vinicius tests two homepage layouts on 150{,}000 visitors each; 9{,}600 sign up for the newsletter with layout B and 9{,}150 with layout A. Find z and the p-value for a two-sided test, and a 95\% interval for p_B - p_A.
Leticia will switch to layout B only if it raises the sign-up rate by at least 0.005. Using the interval from Exercise 1, should she switch? Is the result statistically significant, practically significant, both or neither?
Suppose the layout test had used 3{,}000 visitors per version with the same rates, 192 and 183 sign-ups. Find z and the p-value, and compare with Exercise 1.
For a gap of 0.003 in sign-up rates near a pooled rate of 0.0625, about how many visitors per version make the two-sided test significant at \alpha = 0.05?
A free-delivery banner is shown to 120 visitors, and 30 buy; 120 others see no banner, and 18 buy. Test at \alpha = 0.05, find a 95\% interval, and say what Leticia should conclude.
Murilo’s report says “the new button had a p-value of 0.0001, so it has a large effect”. Explain what the p-value does and does not show.
Study C found a difference of 0.001 with interval 0.0005 to 0.0015 and p-value 0.0001. Study D found a difference of 0.06 with interval -0.01 to 0.13 and p-value 0.09. Which gives stronger evidence of some difference? Which leaves open a larger effect?
A faster server costs 1{,}200 dollars a month. The shop has 100{,}000 visitors a month and makes 3 dollars of profit per order. Find the break-even gain in purchase rate, and decide whether an interval of 0.006 to 0.011 shows that the server pays.

Many tests at once

Murilo runs 50 independent tests at \alpha = 0.05 when no change matters. Find the expected number of false rejections and the chance of at least one.
Repeat Exercise 9 for 10 independent tests at \alpha = 0.01.
With 12 tests and an overall level of 0.05, find the stricter level \alpha / k. Which of the p-values 0.003, 0.008, 0.020 and 0.0004 pass it?
How many independent tests at \alpha = 0.05, all with true nulls, make the chance of at least one false rejection greater than \displaystyle \frac{1}{2}?
Vinicius tested 6 headlines against the original, and one had p-value 0.03. His report mentions only that one. What should an honest report say, and what should happen next?
With 5 independent tests each at \displaystyle \frac{0.05}{5}, all with true nulls, find the chance of at least one false rejection.
Of 30 independent tests at \alpha = 0.05, 4 rejected. If no null were false, find the expected number of rejections and the chance of 4 or more.
Before a test, Murilo names the purchase rate as the main outcome and also plans to look at 9 other outcomes. Explain how this helps, and how the other 9 results should be reported.

Reasoning

Explain why, with a large enough sample, any nonzero true difference is statistically significant.
Explain why a nonsignificant result from a small study is not evidence that the change has no effect.
Explain why testing each of k hypotheses at \alpha / k keeps the chance of any false rejection at or below \alpha, and what it costs.
Explain why k\alpha is the expected number of false rejections even when the tests are not independent, while 1 - (1 - \alpha)^k needs independence.
Explain why a result picked as the best of many should be confirmed on new data rather than retested on the data that picked it.

Answers

\hat{p}_B = 0.064, \hat{p}_A = 0.061, \hat{p}_C = 0.0625. Pooled \text{SE} \approx 0.000884, z \approx 3.39, p-value \approx 0.0007. Unpooled \text{SE} \approx 0.000884, margin \approx 0.0017, interval about 0.0013 to 0.0047.
Not switch: the whole interval lies below 0.005, so the data say the gain is smaller than she needs. The result is statistically significant (it clears 0) but not practically significant.
\hat{p}_C = 0.0625, \text{SE} = 0.00625, z = 0.48, p-value \approx 0.63: fail to reject. The gap is the same; with 50 times fewer visitors the standard error is \displaystyle \sqrt{50} \approx 7.1 times larger.
\displaystyle n = 2(0.0625)(0.9375)\left(\frac{1.96}{0.003}\right)^2 \approx 50{,}000 visitors per version.
\hat{p}_C = 0.2, \text{SE} \approx 0.0516, z \approx 1.94, p-value \approx 0.053: fail to reject. Unpooled \text{SE} \approx 0.0512, interval about -0.0004 to 0.2004. Not convincing evidence, but gains up to 20 points are plausible; she should run a larger test.
It shows the data would be very unlikely if the button made no difference: strong evidence of some difference. It says nothing about size; in a large study a tiny effect gives a tiny p-value. An interval shows the size.
Study C gives far stronger evidence of some difference, but its whole interval is below 0.0015. Study D is not significant, yet its interval reaches 0.13: it leaves open a much larger effect.
Break-even \displaystyle d = \frac{1{,}200}{3 \times 100{,}000} = 0.004. The whole interval lies above 0.004; even at 0.006 the extra profit is 300{,}000(0.006) = 1{,}800 dollars a month, more than the cost, so the server pays.
Expected 50(0.05) = 2.5. P(\text{at least one}) = 1 - 0.95^{50} \approx 0.923.
Expected 10(0.01) = 0.1. P(\text{at least one}) = 1 - 0.99^{10} \approx 0.096.
\displaystyle \frac{0.05}{12} \approx 0.00417. Only 0.003 and 0.0004 pass.
14 tests: 0.95^{13} \approx 0.513 leaves the chance at 0.487, while 0.95^{14} \approx 0.488 makes it 0.512.
It should say that 6 headlines were tested and only one reached 0.03. With 6 tests the chance of at least one false rejection is 1 - 0.95^6 \approx 0.26, and 0.03 does not pass \displaystyle \frac{0.05}{6} \approx 0.0083, so that headline should be retested on new visitors.
1 - 0.99^5 \approx 0.049.
Expected 30(0.05) = 1.5. Binomial with n = 30, p = 0.05: P(0) \approx 0.215, P(1) \approx 0.339, P(2) \approx 0.259, P(3) \approx 0.127, which sum to 0.939, so P(4 \text{ or more}) \approx 1 - 0.939 = 0.061. Unusual, not rare; retest the four.
The main outcome, fixed in advance, keeps its false-alarm rate \alpha. The other 9 are reported as exploratory, with the number of tests stated, and any striking one is confirmed on new data.
The standard error has \displaystyle \sqrt{n} in its denominator, so it falls toward 0 while the observed difference settles near the true one. Their ratio z then grows without bound, so it passes any critical value.
A small study has a large standard error, so its interval is wide and its power is low: even a large real effect often fails to reach significance. Failing to reject means the data cannot rule out zero, not that they rule out an effect.
The chance that at least one of k events happens is at most the sum of their chances, \displaystyle k \cdot \frac{\alpha}{k} = \alpha. The cost is power: each test demands stronger evidence, so real effects are missed more often.
Each test’s false rejection has mean \alpha, and the mean of a sum is the sum of the means, which holds for any dependence. The chance of none equals (1 - \alpha)^k only if the chances multiply, which is what independence means.
The best of many was chosen because it looked extreme, partly by luck, and the same data repeat that luck. New data played no part in the choice, so a test on them keeps its rate \alpha.