p-values and the significance level
A new furnace makes fewer flawed vases in one sample. If the furnace were really no better, how often would a sample look this good by luck alone, and how small does that chance have to be before it counts?
Where you are 8 in Statistical inference
Why this exists
A made-up glassworks in Bohemia has blown vases for years in an old furnace, and 8\% of its vases come out with bubbles trapped in the glass. Zdenka, who runs the works, tries a new furnace that is meant to hold a steadier heat. She takes a random sample of 300 vases from the new furnace, and 15 of them have bubbles:
Her hypotheses were written before she looked (see Null and alternative hypotheses): p is the proportion of all vases from the new furnace that have bubbles, and
The sample came in at 0.05, below 0.08. But samples vary. Even if the new furnace were no better at all, some samples of 300 would land below 0.08 by chance, and a few would land well below it. The question Zdenka needs answered is a number: if the new furnace really makes bubbles in 8\% of its vases, how often would a sample of 300 come out at 0.05 or lower? That number is the p-value. The second question, how small it must be before she takes it seriously, is answered by the significance level.
Definition
The lesson runs over two days. Part 1 defines the p-value and finds it from a test statistic; Part 2 sets the significance level, reads the p-value as a conditional probability, and finds it by simulation.
The p-value. Suppose H_0 is true. The p-value is the probability of getting a statistic at least as extreme as the one actually observed, where “extreme” means far from the null value in the direction (or directions) that H_a names. For a test about one proportion p, with null value p_0, the extreme direction comes from the alternative:
- H_a\colon p < p_0: the p-value is the chance of a sample proportion at or below the observed \hat{p};
- H_a\colon p > p_0: the chance of a sample proportion at or above the observed \hat{p};
- H_a\colon p \neq p_0: the chance of a sample proportion at least as far from p_0, on either side, as the observed \hat{p}.
A small p-value means the observed result would be rare if H_0 were true, so it is evidence against H_0; the smaller the p-value, the stronger that evidence. A large p-value means results like the observed one happen often under H_0, so the data give little or no evidence against it.
The significance level. The significance level, written \alpha, is a cutoff chosen before the data are seen. A p-value at or below \alpha counts as evidence against H_0 strong enough to act on. Common choices are \alpha = 0.05, 0.01 and 0.10; the choice depends on how costly it would be to act on a false alarm.
What the p-value is a probability of. The p-value is a conditional probability: the condition is “H_0 is true”, and the event is “data at least this extreme”. It is computed by assuming H_0. So it is not the probability that H_0 is true, and 1 minus it is not the probability that H_a is true. Those would be probabilities of the hypotheses given the data, with the condition and the event the other way round.
Intuition
When the conditions for a one-proportion test hold (see Conditions for one proportion), the sample proportion, if H_0 is true, follows approximately a normal model with mean p_0 and standard deviation
That curve is the world in which H_0 is true. The observed \hat{p} is a point on its axis, and the test statistic
says how many standard deviations that point sits from p_0. The p-value is the area under the curve beyond the observed point, on the side H_a points to. For Zdenka’s furnace, Figure 1 shades it.
Results right of 0.05 are less extreme than hers and stay unshaded. If the question had been “has the bubble rate changed?”, a sample at 0.11, the same distance above 0.08, would be as strong a sign of change as one at 0.05, and both tails would count, as in Figure 2.
Results
From z to a p-value. With Z standard normal (see The standard normal distribution and Normal calculations: areas and percentiles):
| Alternative | p-value |
|---|---|
| H_a\colon p < p_0 | P(Z \le z) |
| H_a\colon p > p_0 | P(Z \ge z) |
| H_a\colon p \neq p_0 | 2\,P(Z \ge \lvert z \rvert) |
A p-value is between 0 and 1. For a one-sided test, a sample on the wrong side of p_0 (away from H_a) gives a p-value above 0.5.
Smaller means stronger evidence against H_0. A p-value measures evidence against H_0; it never measures evidence for H_0.
The significance level is a false-alarm rate. If H_0 is true and the test is run on many samples, about a share \alpha of them give a p-value at or below \alpha:
A simulated p-value is the share of samples, simulated with H_0 true, whose result is at least as extreme as the observed one. It estimates the same probability as the normal-curve area.
Why it is true
Why doubling works. The normal curve is symmetric about its center. The area beyond \lvert z \rvert on the right equals the area beyond -\lvert z \rvert on the left, so the two tails of a two-sided test together are twice one of them.
Why the false-alarm rate is \alpha. Take \alpha = 0.05. The p-value is at or below 0.05 exactly when the observed z falls in the most extreme 5\% of the null curve, in the direction of H_a. If H_0 is true, the curve describes how z really varies, so z lands in that region in 5\% of samples. The same argument works for any \alpha.
Why \alpha comes first. That guarantee holds only if the cutoff is fixed before the p-value is seen. Someone who chooses \alpha after looking can always pick one just above the p-value, and the cutoff then says nothing about how often a true H_0 is wrongly doubted.
Why it is not P(H_0 \text{ true}). The p-value is P(\text{data this extreme} \mid H_0). The conditional probability formula shows that reversing the condition changes the number:
and the right side depends on how often H_0 is true to begin with and on how likely the data are when H_0 is false, neither of which enters the p-value. Example 6 puts numbers on this.
Worked examples
Examples 1 to 4 belong to Part 1: finding the p-value and reading it. Examples 5 to 7 belong to Part 2: the significance level, what the p-value is a probability of, and p-values by simulation.
The new furnace
Find and interpret the p-value for Zdenka’s test: H_0\colon p = 0.08, H_a\colon p < 0.08, a random sample of 300 vases from the new furnace, 15 with bubbles.
SolutionThe conditions hold: the sample is random, 300 vases are far fewer than 10\% of all the vases the furnace will make, and the expected counts under H_0 are np_0 = 300(0.08) = 24 and n(1 - p_0) = 276, both at least 10. So the normal model for \hat{p} applies.
Under H_0 the standard deviation of \hat{p} uses the null value p_0 = 0.08, since the test works inside the world where H_0 is true:
The observed proportion is \hat{p} = 0.05, so
The alternative is p < 0.08, so the extreme results are those at or below 0.05, the left tail:
(Technology with the unrounded z \approx -1.915 gives 0.0277; a table at -1.92 gives 0.0274. The difference is rounding.)
In context: if the new furnace made bubbles in 8\% of its vases, just like the old one, a random sample of 300 would show a bubble rate of 0.05 or lower only about 2.8\% of the time. That is fairly rare, so the sample is real evidence that the new furnace makes fewer bubbled vases.
The same data, a two-sided question
Vojtech, who runs the cooling ovens, worried before any data came in that the new furnace might change the bubble rate in either direction, since a hotter furnace could also add bubbles. His hypotheses are H_0\colon p = 0.08, H_a\colon p \neq 0.08. Using the same sample, find and interpret his p-value.
SolutionThe statistic is the same, z \approx -1.92. What changes is which results count as extreme. Under H_a\colon p \neq 0.08, a sample at least 0.03 above 0.08 (that is, at 0.11 or more) is as strong a sign of change as one at 0.05 or less. Both tails count, and by symmetry they are equal:
about 0.055. If the bubble rate were still 8\%, a sample of 300 would land at least as far from 0.08 as this one, on one side or the other, about 5.5\% of the time. Same data, twice the p-value, because Vojtech’s question counts surprises in both directions.
A test in the other direction
Bozena buys the sand. She suspected, before testing, that a new supplier’s sand makes more pieces come out cloudy than the usual 12\%. In a random sample of 200 pieces made with the new sand, 31 are cloudy. Find and interpret the p-value.
SolutionWith p the proportion of all pieces made with the new sand that come out cloudy, H_0\colon p = 0.12 and H_a\colon p > 0.12. The expected counts are 200(0.12) = 24 and 176, both at least 10, and the sample is random. Then
The alternative points up, so the p-value is the right tail:
If the new sand were no worse than the old, about 6.4\% of samples of 200 would show a cloudy share of 0.155 or more. That is some evidence against H_0, weaker than in Example 1: a result like this turns up by chance about once in sixteen samples.
Data on the wrong side
A second trial week from the new furnace gives a random sample of 250 vases, 23 with bubbles. With Zdenka’s hypotheses (H_a\colon p < 0.08), find the p-value and say what it means.
SolutionThe expected counts are 250(0.08) = 20 and 230. Then
The sample is above 0.08, the opposite of what H_a claims. The p-value is still the left tail, because “at least as extreme in the direction of H_a” means “at or below 0.092”:
If the furnace were no better, about three samples in four would look at least this good. These data give no evidence that the new furnace makes fewer bubbled vases. They also do not show that it is no better; a p-value never measures evidence for H_0.
Choosing the cutoff first
Before any vases were sampled, the three set their significance levels. Zdenka chose \alpha = 0.05 for her one-sided test. Vojtech chose \alpha = 0.05 for his two-sided test. Bozena argued that replacing a furnace is expensive, so the evidence should be stronger before anyone acts, and asked what Zdenka’s result looks like at \alpha = 0.01. Compare each p-value with its cutoff.
SolutionZdenka: 0.028 \le 0.05, so at her chosen level the result counts as evidence against H_0\colon p = 0.08.
Vojtech: 0.0555 > 0.05, so at his level the same data are not strong enough. The difference from Zdenka comes from the questions they asked before the data, not from the vases.
Bozena’s level: 0.028 > 0.01, so at \alpha = 0.01 the evidence is not strong enough either.
A smaller \alpha demands more, and in exchange a furnace that is really no better is wrongly taken as better less often: tested on many samples, it would give a p-value at or below 0.05 about 5\% of the time, but at or below 0.01 only about 1\%. Saying what to conclude from each comparison is the work of the next lesson.
Reading the condition the right way round
Zdenka reads in a trade paper about 1{,}000 made-up furnace trials at glassworks across the region. In 900 of them the new furnace was really no better (the null hypothesis was true); in 100 it was really better. Each trial ran a test at \alpha = 0.05. Of the 900 trials with H_0 true, 45 gave a p-value at or below 0.05; of the 100 with a better furnace, 60 did. Find P(\text{p-value} \le 0.05 \mid H_0 \text{ true}) and P(H_0 \text{ true} \mid \text{p-value} \le 0.05).
Solution| p-value at or below 0.05 | p-value above 0.05 | Total | |
|---|---|---|---|
| H_0 true (no better) | 45 | 855 | 900 |
| H_0 false (better) | 60 | 40 | 100 |
| Total | 105 | 895 | 1{,}000 |
Given H_0 true, look only at the first row:
This is \alpha, as the Results said it would be. Given a p-value at or below 0.05, look only at the first column:
Among the trials with a small p-value, about 43\% had a furnace that was no better. The first probability is fixed by \alpha; the second depends on how many trials had H_0 true to begin with and on how often better furnaces were caught, which a single p-value knows nothing about. So a p-value of 0.028 does not mean “a 2.8\% chance the new furnace is no better”.
A p-value by simulation
Bozena does not trust the normal model and asks a computer instead. It simulates 1{,}000 random samples of 300 vases from a furnace whose bubble rate is exactly 0.08, the world where H_0 is true, and records the number of bubbled vases in each (Figure 3). Use the simulation to estimate the p-value for Zdenka’s sample of 15 bubbled vases, with H_a\colon p < 0.08.
“At least as extreme in the direction of H_a” means 15 bubbled vases or fewer, the same as \hat{p} \le 0.05. The shaded bars hold 38 of the 1{,}000 simulated samples, so
This is the normal-curve area of Example 1, estimated by counting instead of by formula. The exact chance of 15 or fewer bubbled vases out of 300, at a rate of 0.08, is about 0.029, close to the normal area, so the gap is mostly the simulation’s own chance variation: another run would give a slightly different share. Both say that if the furnace were no better, a sample this good would be uncommon.
The share of reshuffles at least as extreme in comparing treatments by rerandomizing is a p-value of exactly this kind, with the reshuffling playing the part of “H_0 true”.
Boundaries
The p-value assumes H_0. Every p-value is computed inside the world where the null hypothesis is true. It is the probability of the data (or more extreme data) given H_0, not the probability of H_0 given the data, and 1 minus the p-value is not the probability that H_a is true. Finding the probability of a hypothesis needs information the test does not use (Example 6).
The p-value needs its model. A normal-curve p-value is only as good as the conditions behind the normal model: a random sample, independence (the 10\% condition), and expected counts np_0 and n(1 - p_0) of at least 10. When the expected counts are small, simulate instead. No method repairs a sample that was not random; then the p-value describes chance variation that did not produce the data.
\alpha is fixed before the data. A cutoff chosen after seeing the p-value, or a one-sided alternative chosen after seeing which way the data fell, no longer has false-alarm rate \alpha.
A large p-value does not show H_0 is true. It says the data are consistent with H_0. They may be consistent with values near p_0 as well; a small sample can miss a real difference, a risk measured in Type I and Type II errors.
A small p-value does not say how big the effect is. With a large enough sample, a difference too small to matter in practice can give a tiny p-value. The size of the difference is read from \hat{p} - p_0, or better from a confidence interval.
Where this goes next
The p-value and \alpha together give the decision. Deciding and concluding compares the two and turns the comparison into a sentence about the furnace in context. The one-sample z-test for a proportion joins hypotheses, conditions, test statistic, p-value and conclusion into one test, and Hypothesis testing runs the same reasoning through tests for means, differences and slopes. The chance that a test is fooled, by a true H_0 or a false one, is the subject of Type I and Type II errors.
Exercises
Part 1: from a test statistic to a p-value
- A test has H_a\colon p < p_0 and test statistic z = -1.40. Find the p-value.
- A test has H_a\colon p > p_0 and z = 2.15. Find the p-value.
- A test has H_a\colon p \neq p_0 and z = -2.30. Find the p-value.
- A test has H_a\colon p \neq p_0 and z = 0.85. Find the p-value.
- A test has H_a\colon p > p_0 and z = -0.50. Find the p-value and explain why it is so large.
- A test statistic is z = 1.75. Find the p-value for each alternative: (a) p > p_0; (b) p < p_0; (c) p \neq p_0.
Part 1: p-values from data at the glassworks
- Zdenka tries a third furnace design, again testing H_0\colon p = 0.08 against H_a\colon p < 0.08. A random sample of 400 vases has 22 with bubbles. Find z and the p-value, and interpret the p-value in context.
- Historically 5\% of the works’ goblets arrive at shops with a chipped rim. Vojtech tries a new packing crate and wants to know whether the chipped share has changed, in either direction. In a random sample of 500 goblets packed the new way, 34 arrive chipped. State the hypotheses, then find z and the p-value.
- Historically 60\% of the works’ pieces are sold as first quality. Bozena suspects that a new training plan for the glassblowers raises that share. In a random sample of 150 pieces made after the training, 99 are first quality. State the hypotheses, then find z and the p-value.
- Suppose Zdenka’s sample had been 600 vases with 30 bubbled, the same \hat{p} = 0.05 as in Example 1. Find z and the p-value, and compare with Example 1.
- A test of H_0\colon p = 0.08 against H_a\colon p < 0.08 for a fourth furnace gives a p-value of 0.21. Write one sentence interpreting it in context.
Part 2: the significance level and what a p-value means
- For each pair, say whether the p-value is small enough to count as evidence against H_0: (a) p-value 0.041, \alpha = 0.05; (b) p-value 0.041, \alpha = 0.01; (c) p-value 0.12, \alpha = 0.10; (d) p-value 0.008, \alpha = 0.01.
- Vojtech’s test gives a p-value of 0.07. He then says, “I’ll use \alpha = 0.10, so it counts.” What is wrong with this?
- Zdenka’s p-value was 0.028. Say whether each statement is correct, and explain: (a) “There is a 2.8\% chance that the new furnace is no better than the old one.” (b) “If the new furnace were no better, a sample of 300 would show a bubble rate of 0.05 or lower about 2.8\% of the time.” (c) “There is a 97.2\% chance that the new furnace is better.” (d) “The p-value was worked out assuming the new furnace is no better.”
- In 2{,}000 made-up quality tests, 1{,}600 had H_0 true and 400 had H_0 false. Of the 1{,}600, 80 gave a p-value at or below 0.05; of the 400, 240 did. Find P(\text{p-value} \le 0.05 \mid H_0 \text{ true}) and P(H_0 \text{ true} \mid \text{p-value} \le 0.05).
- Over a year the glassworks runs 40 separate tests at \alpha = 0.05, and in every one of them H_0 happens to be true. About how many would you expect to give a p-value at or below 0.05?
Part 2: p-values by simulation
- Zdenka tests H_a\colon p > p_0 for the share of a new colored glass that cracks while cooling. A computer simulates 400 samples with H_0 true, and 17 of them show at least as many cracked pieces as her real sample. Estimate the p-value.
- Vojtech tests a two-sided alternative. Of 800 samples simulated with H_0 true, 21 fall at least as far below the null value as his real sample fell, and 19 at least as far above. Estimate the p-value.
Reasoning
- Explain why the p-value for a two-sided test is twice the one-sided tail area when the normal model is used.
- Write the p-value and the probability that H_0 is true as conditional probabilities, and explain why they are different numbers.
- Explain why the significance level must be chosen before the data are seen.
- A test gives a p-value of 0.40. Does that show H_0 is true? Explain.
- Bozena runs the simulation of Example 7 a second time and gets 33 samples of 1{,}000 at 15 or fewer bubbled vases, not 38. Explain why the two runs differ, and what would make the simulated p-value steadier.
- Explain why a smaller significance level makes a false alarm, treating a true H_0 as false, less likely, and what it costs.
Answers
- P(Z \le -1.40) \approx 0.0808, about 0.081. The alternative points down, so the left tail.
- P(Z \ge 2.15) \approx 0.0158, about 0.016. The right tail.
- 2\,P(Z \le -2.30) \approx 2(0.0107) \approx 0.021. Both tails beyond \lvert z \rvert = 2.30.
- 2\,P(Z \ge 0.85) \approx 2(0.1977) \approx 0.395. A result this close to p_0 is common.
- P(Z \ge -0.50) \approx 0.6915. The sample fell below p_0, the opposite side from H_a, so most of the curve lies on the extreme side of it.
- (a) P(Z \ge 1.75) \approx 0.0401. (b) P(Z \le 1.75) \approx 0.9599. (c) 2(0.0401) \approx 0.0801.
- \displaystyle \hat{p} = \frac{22}{400} = 0.055, \displaystyle \sigma_{\hat{p}} = \sqrt{\frac{0.08 \times 0.92}{400}} \approx 0.0136, z \approx -1.84, p-value = P(Z \le -1.84) \approx 0.033. If the third design made bubbles in 8\% of vases, a sample of 400 would show 0.055 or lower about 3.3\% of the time. (Expected counts 32 and 368.)
- p = the proportion of all goblets packed the new way that arrive chipped. H_0\colon p = 0.05, H_a\colon p \neq 0.05. \displaystyle \hat{p} = \frac{34}{500} = 0.068, \displaystyle \sigma_{\hat{p}} = \sqrt{\frac{0.05 \times 0.95}{500}} \approx 0.00975, z \approx 1.85, p-value = 2\,P(Z \ge 1.85) \approx 0.065. (Expected counts 25 and 475.)
- p = the proportion of all pieces made after the training that are first quality. H_0\colon p = 0.60, H_a\colon p > 0.60. \displaystyle \hat{p} = \frac{99}{150} = 0.66, \displaystyle \sigma_{\hat{p}} = \sqrt{\frac{0.60 \times 0.40}{150}} = 0.04, z = 1.5, p-value = P(Z \ge 1.5) \approx 0.0668.
- \displaystyle \sigma_{\hat{p}} = \sqrt{\frac{0.08 \times 0.92}{600}} \approx 0.0111, z \approx -2.71, p-value \approx 0.0034. The same sample proportion from twice as many vases is much stronger evidence, because a larger sample varies less, so 0.05 is more standard deviations below 0.08.
- If the fourth furnace made bubbles in 8\% of its vases, about 21\% of samples of that size would show a bubble rate at least as low as the one observed; that is common, so the data give little evidence that the fourth furnace is better.
- (a) Yes, 0.041 \le 0.05. (b) No, 0.041 > 0.01. (c) No, 0.12 > 0.10. (d) Yes, 0.008 \le 0.01.
- The cutoff was chosen after seeing the p-value, to make the result count. A cutoff picked that way can always be set just above the p-value, so it no longer controls how often a true H_0 is wrongly doubted. \alpha must be fixed first.
- (a) Wrong: that is the probability of H_0 given the data; the p-value is the probability of the data given H_0. (b) Correct: it is the definition, in context. (c) Wrong, for the same reason as (a). (d) Correct: every p-value is computed with H_0 assumed true.
- Given H_0 true, use the 1{,}600 tests: \displaystyle \frac{80}{1{,}600} = 0.05. Given a p-value at or below 0.05, use the 80 + 240 = 320 such tests: \displaystyle \frac{80}{320} = 0.25.
- About 40 \times 0.05 = 2. With H_0 true, each test gives a p-value at or below 0.05 with probability 0.05.
- \displaystyle \frac{17}{400} = 0.0425, about 0.043.
- Both tails count: \displaystyle \frac{21 + 19}{800} = \frac{40}{800} = 0.05.
- The normal curve is symmetric about p_0, so the area beyond \lvert z \rvert on the right equals the area beyond -\lvert z \rvert on the left, and a two-sided test counts both.
- p-value = P(\text{extreme data} \mid H_0); the other is P(H_0 \mid \text{data}). The condition and the event are swapped. By the conditional probability formula the second depends on how often H_0 is true to begin with and on how likely such data are when H_0 is false, which the p-value does not use (Example 6 and Exercise 15).
- If H_0 is true, a p-value falls at or below a fixed \alpha in a share \alpha of samples. That guarantee needs \alpha fixed in advance; chosen after the data, it can be fitted to the result and controls nothing.
- No. It says data like these are common when H_0 is true, so there is little evidence against it. The same data may be just as common for values of p near p_0, so H_0 is not shown true.
- Each run draws new random samples, so the count at 15 or fewer varies by chance from run to run. More simulated samples, say 10{,}000, make the share vary less.
- If H_0 is true, the chance of a p-value at or below \alpha is \alpha, so a smaller \alpha means fewer false alarms. The cost is that a real difference needs stronger evidence to clear the cutoff, so it is missed more often.