Type I and Type II errors
A test on a sample of brake cables can send back a good shipment or let a bad one through. How often does each mistake happen, and which one should the factory fear more?
Where you are 7 in Hypothesis testing
Why this exists
A made-up bicycle factory in the Netherlands buys brake cables by the shipment. Its rule is that a shipment is acceptable if at most 2\% of its cables are faulty. Joris, who runs incoming inspection, cannot test every cable, so from each shipment he takes a random sample of 500 and tests
where p is the proportion of faulty cables in the whole shipment, at significance level \alpha = 0.05. A p-value is the chance, if H_0 is true, of a sample proportion at least as extreme as the one observed; Joris rejects H_0 when the p-value is at or below \alpha, and then the shipment goes back to the supplier.
The decision rests on 500 cables, not on the whole shipment, so it can be wrong. A good shipment can, by bad luck, give a sample with many faulty cables and be sent back. A bad shipment can give a sample that looks fine and be put on the assembly line. These are two different mistakes with two different costs. This lesson names them, says how often each one happens, and shows how the choice of \alpha trades one against the other.
Definition
The lesson runs over two days. Part 1 names the two errors, finds the chance of the first, and weighs their consequences. Part 2 computes the chance of the second at a specific alternative.
The four outcomes. The truth is either “H_0 true” or “H_0 false”, and the test’s decision is either “reject H_0” or “fail to reject H_0”. Crossing them gives four outcomes, two right and two wrong:
| Fail to reject H_0 | Reject H_0 | |
|---|---|---|
| H_0 true | correct decision | Type I error |
| H_0 false | Type II error | correct decision |
A Type I error is rejecting H_0 when H_0 is true. A Type II error is failing to reject H_0 when H_0 is false.
Their probabilities. If H_0 is true, the probability of a Type I error is the significance level \alpha. If H_0 is false, the probability of a Type II error is written \beta, and it is computed at one particular value of the parameter in H_a:
The probability of correctly rejecting a false H_0 at that value, 1 - \beta, is the power of the test there; it is the subject of the next lesson.
Intuition
For a one-proportion z-test (see One sample test for a proportion), “reject when the p-value is at most \alpha” is the same as “reject when \hat{p} lands beyond a cutoff”. Under H_0 the sample proportion is approximately normal with mean p_0 and standard deviation \displaystyle \sigma_{\hat{p}} = \sqrt{p_0(1 - p_0)/n} (see Sampling distribution of a sample proportion). For Joris’s upper-tailed test the p-value is at most 0.05 exactly when the test statistic z is at least 1.645, so the cutoff for \hat{p} is
A sample with more than 0.0303 of its cables faulty (in counts, 16 or more of 500) sends the shipment back.
Now picture two shipments. One is exactly 2\% faulty, so \hat{p} follows the solid curve in Figure 1. The other is 4\% faulty, so \hat{p} follows the dashed curve, centered at 0.04. The same cutoff cuts both. Under the solid curve, the small area to the right is the chance of sending back a good shipment: that is \alpha. Under the dashed curve, the area to the left is the chance of keeping a bad one: that is \beta.
The curves overlap, and that overlap is why a test can be wrong at all. Sliding the cutoff right shrinks the first shaded area and grows the second; sliding it left does the opposite. With the sample size fixed, no position of the cutoff makes both small at once.
Results
The chance of a Type I error is \alpha. If H_0 is true and its conditions hold, a test at level \alpha rejects H_0 with probability \alpha (approximately, when the test rests on a normal model).
A Type II error has no single probability. \beta depends on the true value of the parameter. For an upper-tailed test of one proportion with cutoff c,
computed with \hat{p} approximately normal with mean p_1 and standard deviation \displaystyle \sqrt{p_1(1 - p_1)/n}. For a lower-tailed test, \beta = P(\hat{p} \ge c \mid p = p_1).
\beta shrinks as the true value moves away from p_0. A shipment far worse than 2\% is hard to miss; one barely worse is easy to miss.
\alpha and \beta pull against each other. With the sample size and the alternative value fixed, a smaller \alpha gives a larger \beta, and a larger \alpha a smaller \beta.
Only one error is possible after a decision. A test that rejects H_0 may have made a Type I error and cannot have made a Type II error; a test that fails to reject may have made a Type II error and cannot have made a Type I error.
Choose \alpha by the costs. If a Type I error is the more serious mistake, choose a small \alpha such as 0.01; if a Type II error is the more serious, a larger \alpha such as 0.10 lowers \beta.
Why it is true
Why the Type I chance is \alpha. Suppose p = p_0. Then z follows (approximately) the standard normal curve, and Joris rejects when z \ge 1.645, an event of probability P(Z \ge 1.645) = 0.05. The cutoff 1.645 was chosen for exactly this reason. For any \alpha the critical value is chosen so that the area beyond it under the null curve is \alpha.
Why \beta depends on p. The cutoff is fixed by H_0 alone. But how often \hat{p} falls short of it depends on where \hat{p} really centers, which is the true p. A true p just above 0.02 puts the center of the dashed curve just right of 0.02, mostly below c; a true p well above c puts most of the curve beyond it. So “H_0 is false” does not give one number; only a specific alternative value does.
Why lowering \alpha raises \beta. A smaller \alpha needs a larger critical value, so the cutoff c moves right. The alternative curve does not move, so more of it lies left of c, and \beta, the area to the left, grows.
Worked examples
Examples 1 to 4 belong to Part 1: the two errors, their consequences, and the chance of a Type I error. Examples 5 to 7 belong to Part 2: computing \beta and seeing how it changes.
The two errors at the factory
Describe a Type I error and a Type II error for Joris’s test of H_0\colon p = 0.02 against H_a\colon p > 0.02, and give a consequence of each.
SolutionA Type I error rejects a true H_0. Here: the shipment really has at most 2\% faulty cables, but the sample has so many faulty ones that Joris concludes the rate is above 2\% and sends the shipment back. The consequence is a good shipment returned: a delay on the assembly line while a replacement comes, shipping costs, and a supplier who was wrongly blamed.
A Type II error fails to reject a false H_0. Here: the shipment really has more than 2\% faulty cables, but the sample looks acceptable, so Joris does not find convincing evidence and the shipment is used. The consequence is that faulty brake cables go onto bicycles that are sold.
The two descriptions use the parameter in words (“the shipment’s proportion of faulty cables”), and each says what is true and what the test decides. That is what makes a description in context complete.
Which error could have happened
Two shipments arrive in the same week. In the sample of 500 from the first, 19 cables are faulty; in the sample from the second, 12. Joris uses the cutoff c \approx 0.0303. For each shipment, give the decision and say which error is possible.
SolutionFirst shipment: \displaystyle \hat{p} = \frac{19}{500} = 0.038, above 0.0303, so Joris rejects H_0 and sends it back. A rejection is wrong only if H_0 was true, so the possible error is a Type I error. A Type II error is impossible here, because a Type II error is a failure to reject.
Second shipment: \displaystyle \hat{p} = \frac{12}{500} = 0.024, at or below 0.0303, so Joris fails to reject H_0 and the shipment is used. The possible error is a Type II error: the shipment may really be more than 2\% faulty. A Type I error is impossible here.
Neither decision tells Joris whether an error was made. The test controls how often each error happens over many shipments, never whether this one was right.
The long-run rate of Type I errors
Over a season the factory receives 40 shipments, and suppose every one of them is in truth exactly 2\% faulty. About how many will Joris send back? Explain why \alpha is the chance of a Type I error.
SolutionThe conditions hold for each sample: the cables are a random sample, 500 is far fewer than 10\% of a shipment, and the expected counts under H_0 are np_0 = 500(0.02) = 10 and n(1 - p_0) = 490, both at least 10. So z is approximately standard normal for a shipment with p = 0.02, and
Each good shipment is sent back with probability 0.05, so about 40 \times 0.05 = 2 of the 40 are sent back, every one of them a Type I error. None of the 40 can produce a Type II error, because H_0 is true for all of them.
The count 10 sits exactly at the large-counts boundary, so the normal model here is adequate but rough. Worked from the exact binomial counts, the chance that a 2\% shipment gives 16 or more faulty cables is about 0.047, a little below 0.05.
Choosing alpha by the costs
Thijs, who buys the cables, says that sending back a good shipment costs money and a day of production. Ineke, who is responsible for safety, says that a shipment with too many faulty brake cables puts riders at risk. (a) Which error is more serious for the cable test, and what does that suggest for \alpha? (b) The factory is also deciding whether to replace its welding robots, testing H_0: the new robots’ weld failure rate equals the old rate, against H_a: it is lower. The new robots are very expensive and the old welds already meet the standard. Which error is more serious there, and what \alpha does that suggest?
Solution(a) A Type I error costs money and time; a Type II error puts faulty brakes on bicycles. The Type II error is the more serious. A larger \alpha, such as 0.10, rejects H_0 on weaker evidence, which lowers \beta; Ineke would argue for it. Thijs’s cost is real, and the price of \alpha = 0.10 is that about 10\% of good shipments get sent back instead of 5\%.
(b) A Type I error here means concluding the new robots are better when they are not, and buying them for nothing. A Type II error means missing a real improvement, while keeping welds that already meet the standard. The Type I error is the more serious, so a small \alpha, such as 0.01, fits: the factory should spend that money only on strong evidence.
The same factory chose differently in the two tests because the consequences differ, not the mathematics. The choice is made before any data are collected.
Computing beta at one alternative
Suppose a shipment is really 4\% faulty. Find \beta for Joris’s test, the probability that the shipment is not sent back.
SolutionThe cutoff comes from H_0, because the rejection rule is set in the world where p = 0.02:
The shipment is kept when \hat{p} \le 0.0303. Now the chance of that is worked out in the world where p = 0.04, so the sampling distribution of \hat{p} is centered at 0.04 with its own standard deviation:
The normal model applies there too: 500(0.04) = 20 and 500(0.96) = 480. The cutoff is
standard deviations below the center, so
(a table at -1.11 gives 0.1335). This is the dashed area left of the cutoff in Figure 1. In context: if a shipment is really 4\% faulty, twice the acceptable rate, this test lets it through about 13\% of the time, roughly one such shipment in eight. The power at p = 0.04 is 1 - 0.134 \approx 0.87.
Beta as the true p changes
Find \beta for the same test when the shipment is really 3\% faulty, and when it is 5\% faulty. Compare with Example 5.
SolutionThe cutoff stays at 0.0303, since it depends only on H_0, \alpha and n. Only the world changes.
If p = 0.03: the standard deviation is \displaystyle \sqrt{0.03 \times 0.97 / 500} \approx 0.00763, and
If p = 0.05: the standard deviation is \displaystyle \sqrt{0.05 \times 0.95 / 500} \approx 0.00975, and
| True p | 0.03 | 0.04 | 0.05 |
|---|---|---|---|
| \beta | 0.516 | 0.134 | 0.022 |
A 3\% shipment sits so close to the cutoff that the test misses it about half the time; a 5\% shipment is missed only about 2\% of the time. The farther the truth lies from p_0, the farther the alternative curve sits beyond the cutoff, and the less of it is left behind.
Beta when alpha changes
Thijs proposes \alpha = 0.01 to send back fewer good shipments. Find the new cutoff and \beta at p = 0.04, and compare with \alpha = 0.05.
SolutionThe upper 1\% point of the standard normal curve is z^* \approx 2.326, so
At p = 0.04 the standard deviation is still 0.00876:
Lowering \alpha from 0.05 to 0.01 cuts the Type I chance to a fifth, and roughly doubles \beta at p = 0.04, from 0.134 to 0.268: about one 4\% shipment in four would now be used. Since Example 4 judged the Type II error the more serious one for brake cables, Thijs’s proposal moves the wrong way for safety. Lowering both errors at once takes a change other than \alpha, which is the subject of Power and what raises it.
Boundaries
\alpha is the Type I chance only when p = p_0 exactly. Joris’s H_0 stands for “at most 2\%”. A shipment that is really 1\% faulty is sent back less often than 5\% of the time, so \alpha is the largest Type I chance, reached at p_0.
Each number needs its model. The \alpha and every \beta above come from a normal model for \hat{p}, which needs a random sample, independence, and large expected counts: np_0 and n(1 - p_0) for the cutoff (see Conditions for one proportion), and np_1 and n(1 - p_1) for \beta. Without them the stated error rates are not the real ones. Even with them, the normal answers are approximations. Joris’s np_0 = 10 is right at the edge, and the count of faulty cables is a whole number, so the test really sends a shipment back at 16 or more. Exact binomial counts give \alpha \approx 0.047, and \beta \approx 0.568, 0.151 and 0.020 at p = 0.03, 0.04 and 0.05, against the normal 0.516, 0.134 and 0.022. The pattern is the same; the third decimal place is not to be trusted.
\beta needs a specific alternative. “The probability of a Type II error” with no value of p named has no answer.
The probabilities describe the test, not one decision. \alpha = 0.05 does not mean a rejected shipment has a 5\% chance of being good. It is a chance computed assuming H_0 is true, before the sample is drawn.
\alpha is set in advance. An \alpha chosen after the data no longer has the Type I rate it names.
Where this goes next
Power, 1 - \beta, turns the question around: how likely is the test to catch a shipment that really is bad, and what besides \alpha makes that chance larger. The same two errors run through every test in Hypothesis testing, for means, differences and slopes, and they explain why a significant result and a large one are different things.
Exercises
Part 1: the two errors in context
The factory also buys saddles. A shipment of saddles is acceptable if at most 5\% have torn stitching, and Ineke tests H_0\colon p = 0.05 against H_a\colon p > 0.05, sending the shipment back if she rejects H_0. Exercises 1 and 2 use this test.
- Describe a Type I error and a Type II error for the saddle test, in context.
- Give a consequence of each error for the saddle test. Which matters more to a rider?
- For Joris’s cable test, name the outcome (correct, Type I or Type II): (a) the shipment is 2\% faulty and is sent back; (b) it is 6\% faulty and is used; (c) it is 6\% faulty and is sent back; (d) it is 2\% faulty and is used.
- A shipment of cables is in truth exactly 2\% faulty. With Joris’s test at \alpha = 0.05, what is the probability it is sent back? What is the probability of a Type II error for it?
- Over a year, 80 shipments that are each exactly 2\% faulty are tested at \alpha = 0.05. About how many are sent back?
- Of 120 shipments tested in a year at \alpha = 0.05, 90 are exactly 2\% faulty and 30 are faulty at a rate where \beta = 0.20. About how many Type I errors and how many Type II errors would you expect?
- Choose \alpha = 0.01 or \alpha = 0.10 for each test and explain: (a) H_0: a new handlebar grip lasts as long as the current one, H_a: it lasts longer, where switching means a costly long-term contract and the current grip already wears well; (b) H_0: a batch of tire valves leaks at the accepted rate, H_a: it leaks more, where leaking valves cause flat tires on new bicycles.
- Thijs says, “We tested at \alpha = 0.05 and rejected, so there is a 5\% chance this shipment is actually fine.” What is wrong with this?
- Joris fails to reject H_0 for a shipment of chains. Which error could he have made? Could he have made the other one?
Part 2: computing beta
Exercises 10 to 13 use Joris’s cable test: n = 500, H_0\colon p = 0.02, H_a\colon p > 0.02.
- With \alpha = 0.05 (cutoff 0.0303), find \beta if the shipment is really 3.5\% faulty.
- With \alpha = 0.05, find \beta if the shipment is really 4.5\% faulty.
- Use Exercise 11 to find the power of the test at p = 0.045, and say what it means.
- With \alpha = 0.10, find the cutoff and \beta at p = 0.04.
- Ineke’s saddle test uses a random sample of 400 saddles at \alpha = 0.05. Find the cutoff for \hat{p}, and \beta if the shipment really has 8\% with torn stitching.
- A new paint is meant to chip less. Joris tests H_0\colon p = 0.10 against H_a\colon p < 0.10, where p is the proportion of frames whose paint chips in a drop test, with a random sample of 300 frames at \alpha = 0.05. Find the cutoff, and \beta if the new paint really chips on 6\% of frames.
- Thijs tests whether a new spoke supplier’s share of bent spokes differs from 10\%: H_0\colon p = 0.10, H_a\colon p \neq 0.10, a random sample of 400 spokes, \alpha = 0.05. Find both cutoffs, and \beta if the share is really 14\%.
- Without computing, put \beta for Joris’s test at p = 0.025, p = 0.04 and p = 0.06 in order from largest to smallest, and explain.
- Ineke works out P(\hat{p} \le 0.0303 \mid p = 0.02) \approx 0.95 and calls it \beta. Explain why that is not a Type II error probability.
Reasoning
- Explain why the probability of a Type I error equals \alpha.
- Explain why a test has no single probability of a Type II error.
- With n and the alternative value fixed, explain why lowering \alpha raises \beta.
- Explain why a test that rejects H_0 cannot have made a Type II error.
- In computing \beta, the cutoff uses p_0 in its standard deviation but the final probability uses p_1. Explain why each uses the value it does.
Answers
- Type I: the shipment really has at most 5\% torn saddles, but the sample shows so many that Ineke concludes the rate is above 5\% and sends it back. Type II: the shipment really has more than 5\% torn saddles, but the sample does not give convincing evidence, so it is used.
- Type I: a good shipment is returned, costing time, shipping and a supplier wrongly blamed. Type II: bicycles go out with saddles whose stitching tears. The Type II error matters more to a rider, though torn stitching is a nuisance rather than a danger like a faulty brake.
- (a) Type I error. (b) Type II error. (c) Correct. (d) Correct.
- 0.05, since H_0 is true and the Type I chance is \alpha. The Type II chance is 0: a Type II error needs H_0 to be false.
- About 80 \times 0.05 = 4, every one a Type I error.
- Type I: about 90 \times 0.05 = 4.5. Type II: about 30 \times 0.20 = 6.
- (a) \alpha = 0.01: a Type I error signs a costly contract for a grip that is no better, while a Type II error only keeps a grip that already wears well, so the Type I error is the more serious. (b) \alpha = 0.10: a Type II error sends leaking valves onto new bicycles, the more serious error, and a larger \alpha lowers \beta.
- \alpha is the chance of rejecting if the shipment is fine, P(\text{reject} \mid H_0), worked out before sampling. It is not the chance that H_0 is true given the rejection.
- A Type II error: the shipment may really be more than the accepted rate. He cannot have made a Type I error, which requires rejecting H_0.
- Standard deviation at p = 0.035: \displaystyle \sqrt{0.035 \times 0.965 / 500} \approx 0.00822; \displaystyle z = \frac{0.0303 - 0.035}{0.00822} \approx -0.57; \beta = P(Z \le -0.57) \approx 0.284.
- Standard deviation \displaystyle \sqrt{0.045 \times 0.955 / 500} \approx 0.00927; \displaystyle z = \frac{0.0303 - 0.045}{0.00927} \approx -1.59; \beta \approx 0.056.
- Power = 1 - 0.056 \approx 0.944: a shipment that is really 4.5\% faulty is sent back about 94\% of the time.
- z^* \approx 1.282, c = 0.02 + 1.282(0.00626) \approx 0.0280; \displaystyle z = \frac{0.0280 - 0.04}{0.00876} \approx -1.37; \beta \approx 0.086. Smaller than at \alpha = 0.05.
- \displaystyle \sigma_{\hat{p}} = \sqrt{0.05 \times 0.95 / 400} \approx 0.0109, c = 0.05 + 1.645(0.0109) \approx 0.0679. At p = 0.08: standard deviation \displaystyle \sqrt{0.08 \times 0.92 / 400} \approx 0.0136, \displaystyle z = \frac{0.0679 - 0.08}{0.0136} \approx -0.89, \beta \approx 0.187. (Expected counts 20 and 380 under H_0, 32 and 368 at 0.08.)
- \displaystyle \sigma_{\hat{p}} = \sqrt{0.10 \times 0.90 / 300} \approx 0.0173; lower-tailed, so c = 0.10 - 1.645(0.0173) \approx 0.0715, reject when \hat{p} \le 0.0715. A Type II error is \hat{p} > 0.0715: at p = 0.06 the standard deviation is \displaystyle \sqrt{0.06 \times 0.94 / 300} \approx 0.0137, z \approx 0.84, \beta = P(Z \ge 0.84) \approx 0.201.
- \displaystyle \sigma_{\hat{p}} = \sqrt{0.10 \times 0.90 / 400} = 0.015; cutoffs 0.10 \pm 1.96(0.015), that is 0.0706 and 0.1294. A Type II error is \hat{p} between them. At p = 0.14 the standard deviation is \displaystyle \sqrt{0.14 \times 0.86 / 400} \approx 0.0173, z values \approx -4.00 and \approx -0.61, so \beta \approx 0.271 - 0.000 = 0.271.
- p = 0.025 largest, then 0.04, then 0.06. The cutoff is fixed; the farther the true p lies beyond it, the less of its sampling distribution falls short of it.
- At p = 0.02, H_0 is true, so failing to reject is the correct decision, not an error. That 0.95 is 1 - \alpha. \beta is computed at a value of p where H_0 is false.
- The cutoff is placed so that the area beyond it under the null sampling distribution is \alpha. If H_0 is true, that distribution describes the statistic, so it falls beyond the cutoff, and H_0 is rejected, with probability \alpha.
- \beta is the chance the statistic falls short of a fixed cutoff, and where the statistic’s distribution sits depends on the true parameter value. Each alternative value gives its own \beta.
- A smaller \alpha needs a larger critical value, so the cutoff moves away from p_0, toward the alternative. The alternative’s distribution stays put, so more of it falls on the fail-to-reject side.
- A Type II error is, by definition, failing to reject a false H_0. A test that rejects did not fail to reject, so only a Type I error is possible.
- The cutoff is set by how \hat{p} varies if H_0 is true, so it uses p_0. \beta is the chance of landing short of the cutoff when p = p_1, and then \hat{p} varies with standard deviation \displaystyle \sqrt{p_1(1 - p_1)/n}.