Statistics

Confidence level and interpreting an interval

A computed interval either contains the true value or it does not. So what is the “95%” in a 95% confidence interval a statement about, and what can a report honestly say from one interval?

You need

Point estimates and the margin of errorStatistics

Unlocks

Critical valueStatistics Deciding and concluding from a p-valueStatistics

Where you are 4 in Confidence intervals

Statistics7

Statistical inference8

Confidence intervals4

Confidence level and interpreting an interval

Point estimates and the margin of error Standard error Critical value

Why this exists

Kaisa works for a state wildlife agency that tracks invasive zebra mussels. The state has 12{,}000 lakes, far too many to inspect, so the agency inspects a random sample of 250 and finds mussels in 60 of them (made-up data). As in Point estimates and the margin of error, the estimate is the sample proportion give or take about 2 standard errors:

\hat{p} = \frac{60}{250} = 0.24, \qquad 0.24 \pm 0.054

roughly 0.186 to 0.294, and the agency’s report calls this a 95\% confidence interval.

That percentage hides a real puzzle. The true share p of the state’s lakes with mussels is one fixed number. The interval is two fixed numbers. Either p lies between them or it does not, and nothing about that is left to chance. So “95\%” cannot be the chance that this particular interval is right, the way a forecast gives a chance of rain tomorrow. It is a statement about something else, and a report that puts it in the wrong place claims more than the data support. Part 1 of this lesson pins down what the confidence level measures. Part 2 turns to what one interval lets the agency say and decide.

Definition

A confidence interval for a parameter is a range of values computed from a random sample by a fixed method. In this lesson the method is

\hat{p} \pm 2\,\text{SE}_{\hat{p}}, \qquad \text{SE}_{\hat{p}} = \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}

with the Standard error built from the sample.

The confidence level C of the method is the proportion of all possible random samples for which the interval the method produces captures the parameter. For \hat{p} \pm 2\,\text{SE} from a large random sample, C is about 95\%. The level belongs to the method, and it is fixed before any sample is drawn.

What a computed interval says:

It lists the values of the parameter that are plausible in light of this sample.
Its level says how reliable the method is: across all the random samples that could have been drawn, about C of the intervals capture p. “We are 95\% confident” means “this interval came from a method that captures p for about 95\% of samples”.

What it does not say:

It does not say that p lies in this interval with probability 0.95. Once the interval is computed, p is inside it or it is not.
It is about one parameter, not about individuals. It does not say that 95\% of lakes, or of anything else in the population, fall in the range.
It is not a range for future sample proportions. Another random sample would give its own \hat{p} and its own interval, centered somewhere else.

Intuition

No one can watch the method work on the real lakes, because no one knows p there. A computer can. Siiri builds a made-up state where exactly 25\% of lakes have mussels, so that p = 0.25 is known, and draws 50 random samples of 250 lakes. From each sample she computes \hat{p} \pm 2\,\text{SE}, exactly as the agency would.

Figure 1Fifty simulated random samples of 250 lakes from a made-up state where exactly 25\% of lakes have mussels, each turned into the interval \hat{p} \pm 2\,\text{SE}. The centers wander; the line p = 0.25 stays put. Forty-seven intervals capture it, and the three dashed ones miss.

Each interval in Figure 1 is centered on its own \hat{p}. The centers wander from sample to sample, which is the Sampling distribution of a sample proportion at work (seen by simulation in Simulating a sampling distribution and a margin of error). The widths differ a little too, since each standard error uses its own \hat{p}. The vertical line at p = 0.25 never moves. Forty-seven intervals cross it; three, samples 24, 29 and 36, miss.

An interval misses exactly when its center lands more than about 2 standard errors from p. Most centers land close, because the sampling distribution of \hat{p} is centered at p and is close to normal for samples this large (The central limit theorem), and about 95\% of a normal distribution lies within 2 standard deviations of its center. So about 5 intervals in every 100 miss. In this batch the share captured is \displaystyle \frac{47}{50} = 94\%; another batch of 50 would give a slightly different count, and over thousands of samples the share settles near 95\%.

The real agency sees one row of such a picture, from a state whose vertical line is hidden. Kaisa cannot tell whether her interval is one of the many hits or one of the few misses. What she knows is the method’s record, and that record is what her 95\% reports.

Results

The level is the method’s capture rate. If random samples are drawn over and over and the interval is computed by the same method each time, the share of intervals that capture p approaches C. With a large random sample, \hat{p} \pm 1\,\text{SE} captures about 68\% of the time, \hat{p} \pm 2\,\text{SE} about 95\%, and \hat{p} \pm 3\,\text{SE} about 99.7\%.

An interval in context names three things: the parameter, the population, and the level of the method. For the lakes: “We are 95\% confident that the interval from 0.186 to 0.294 captures the true proportion of all the state’s lakes that have zebra mussels.”

Judging a claimed value. A claimed value of the parameter that lies inside the interval is plausible at that level. A value outside it is not plausible at that level. A value inside is never shown to be the true one; every value in the interval remains a candidate.

Confidence costs width. From the same sample, a higher level needs a larger multiplier and so gives a wider interval. At the same level, a larger sample gives a narrower interval: four times the sample halves the width.

Why it is true

Why the capture rate is about 95\%. The interval \hat{p} \pm 2\sigma_{\hat{p}} reaches p exactly when |\hat{p} - p| \le 2\sigma_{\hat{p}}, because distance on a number line is the same in both directions. For a large random sample the sampling distribution of \hat{p} is close to normal, with center p and standard deviation \displaystyle \sigma_{\hat{p}} = \sqrt{p(1 - p)/n}, so about 95\% of samples satisfy that inequality. The working method uses \text{SE}_{\hat{p}} in place of \sigma_{\hat{p}}, which needs the unknown p; for a large sample the two are close, and the capture rate stays near 95\%.

Why the probability belongs before the sample. Before the lakes are chosen, \hat{p} is a random variable, so the interval is a random interval, and “the interval will capture p” is an event with probability about 0.95. After the lakes are chosen, the interval is two numbers and the event has either happened or not. A coin about to be tossed lands heads with probability \displaystyle \frac{1}{2}; a coin already lying under a hand is heads or tails, and the \displaystyle \frac{1}{2} describes the toss, not the coin.

Why more confidence means more width. To capture p in a larger share of samples, the interval has to reach p even from centers that landed farther away, so the multiplier on \text{SE} grows. The width is 2 \times (\text{multiplier}) \times \text{SE}, so it grows with the multiplier. The standard error has \displaystyle \sqrt{n} in its denominator, which is why only a larger sample narrows an interval without lowering its level.

Worked examples

The lesson is taught in two parts. Examples 1 to 3 belong to the first: what the confidence level measures, read from the method and from many intervals. Examples 4 to 7 belong to the second: interpreting one interval in context, judging claimed values, and weighing confidence against width.

Example 1

The lake interval and what its level describes

The agency’s random sample of 250 lakes has 60 with zebra mussels. Compute the interval \hat{p} \pm 2\,\text{SE} and say what its level of about 95\% describes.

Solution

The parameter is p, the proportion of all 12{,}000 lakes in the state that have mussels, and its point estimate is \displaystyle \hat{p} = \frac{60}{250} = 0.24. The sample is under a tenth of the lakes, 250 \le 1{,}200, so the inspections are close to independent (The ten percent condition), and with 60 lakes with mussels and 190 without, the sampling distribution of \hat{p} is close to normal. The standard error is

\begin{aligned} \text{SE}_{\hat{p}} &= \sqrt{\frac{0.24 \times 0.76}{250}} = \sqrt{\frac{0.1824}{250}} \\ &= \sqrt{0.0007296} \approx 0.0270 \end{aligned}

so the margin is 2 \times 0.0270 \approx 0.054 and the interval is

\begin{aligned} 0.24 - 0.054 &= 0.186 \\ 0.24 + 0.054 &= 0.294 \end{aligned}

The 95\% describes the method. If the agency could draw every possible random sample of 250 lakes and compute \hat{p} \pm 2\,\text{SE} from each, about 95\% of those intervals would capture p. It does not say that this one interval, 0.186 to 0.294, has a 95\% chance of being right; this one is right or wrong already, and no one knows which.

Example 2

Reading the fifty intervals

In Figure 1, (a) what share of the intervals capture p = 0.25? (b) Sample 29 had 45 lakes with mussels. Compute its interval and explain why it misses. (c) How many standard deviations of the sampling distribution separate its \hat{p} from p?

Solution

(a) Forty-seven of fifty capture, \displaystyle \frac{47}{50} = 0.94, close to the method’s 95\%.

(b) Here \displaystyle \hat{p} = \frac{45}{250} = 0.18 and

\begin{aligned} \text{SE}_{\hat{p}} &= \sqrt{\frac{0.18 \times 0.82}{250}} = \sqrt{0.0005904} \\ &\approx 0.0243 \end{aligned}

so the interval is 0.18 \pm 0.0486, from 0.131 to 0.229. Its upper end falls short of 0.25, so it misses. Nothing went wrong in this sample: it is random, the arithmetic is correct, and its center simply landed in the low tail.

\sigma_{\hat{p}} = \sqrt{\frac{0.25 \times 0.75}{250}} = \sqrt{0.00075} \approx 0.0274

and \hat{p} sits 0.25 - 0.18 = 0.07 below p, which is \displaystyle \frac{0.07}{0.0274} \approx 2.56 standard deviations. Centers more than about 2 standard deviations out make up about 5\% of all samples, and nearly all of them produce a miss. Siiri can mark sample 29 only because she knows p. An agency holding this sample would report 0.131 to 0.229 and never learn that it missed.

Example 3

Forty agencies

Suppose 40 state agencies, each with its own state and its own independent random sample, each report a 95\% interval for the share of their lakes with mussels. How many intervals should be expected to capture the true share? What is the probability that all 40 do?

Solution

Before its sample is drawn, each agency’s interval captures its state’s p with probability 0.95. The expected number of captures is 40 \times 0.95 = 38, so about 2 intervals miss. The samples are independent, so the probability that every one captures is the product

0.95^{40} \approx 0.1285

There is about an 87\% chance that at least one agency publishes an interval that misses, and that agency has no way to tell. The confidence level is a track record of the method across many uses, not a guarantee about any one result.

Example 4

Three sentences about one interval

Matti, Siiri and Kaisa each draft the report’s sentence about 0.186 to 0.294. Matti: “There is a 95\% probability that the true share is between 0.186 and 0.294.” Siiri: “If we inspected another random sample of 250 lakes, its sample proportion would be between 0.186 and 0.294 with probability 0.95.” Kaisa: “We are 95\% confident that between 18.6\% and 29.4\% of all the state’s lakes have zebra mussels: the method behind this interval captures the true share for about 95\% of random samples.” Decide what each sentence claims and whether the interval supports it.

Solution

Kaisa’s sentence is supported. It names the parameter (the share with mussels), the population (all the state’s lakes), and places the 95\% on the method.

Matti’s sentence puts a probability on p, which is a fixed number. After the sample, p is in the interval or not, so the probability is 1 or 0 and is unknown; the 0.95 belonged to the method before the sample.

Siiri’s sentence is about future values of \hat{p}. A new sample proportion is centered on p, not on this sample’s 0.24. If p sits near one end of the interval, a new \hat{p} lands inside it far less often than 95\% of the time. The interval estimates p; it is not a forecast of the next sample.

Example 5

From a share to a count of lakes

Express the agency’s interval as a number of the state’s 12{,}000 lakes, and interpret it in context.

Solution

Every lake share times 12{,}000 is a count of lakes, so each endpoint scales:

\begin{aligned} 0.186 \times 12{,}000 &= 2{,}232 \\ 0.294 \times 12{,}000 &= 3{,}528 \end{aligned}

We are 95\% confident that between about 2{,}230 and 3{,}530 of the state’s 12{,}000 lakes have zebra mussels. The level is unchanged: multiplying by the fixed number 12{,}000 moves both endpoints and the true count together, so an interval captures the count exactly when it captures the share. The rounding keeps the honest precision of a sample of 250.

Example 6

Judging two claims

A fishing club claims that 30\% of the state’s lakes now have mussels. A lake association claims the share is only 20\%. What does the 95\% interval say about each claim? Then compare with the interval at 3 standard errors.

Solution

Figure 2 shows the interval and both claims.

Figure 2The agency’s interval from 60 of 250 lakes (made-up data), with two claimed values. The claim 0.20 is among the plausible values; the claim 0.30 is not, at about 95\% confidence.

The value 0.30 lies above the upper end 0.294, so it is not plausible at about 95\% confidence: the data point to a share below 30\%. The value 0.20 lies inside, so it is plausible. That does not show the share is 20\%; 0.22, 0.25 and 0.28 are plausible too, and the data cannot pick among them.

At 3 standard errors, about 99.7\% confidence, the margin is 3 \times 0.0270 \approx 0.081:

0.24 \pm 0.081, \text{ from } 0.159 \text{ to } 0.321

and now 0.30 is inside. The club’s claim is just outside the 95\% interval and inside the wider one, so the verdict on it depends on the level chosen. A claim far outside both, such as 0.40, would be ruled out at either level. Choosing the level before looking at the claim, and then reading the verdict, is the logic later made formal in Deciding and concluding from a p-value.

Example 7

Confidence against width

Find the width of the lake interval at 1, 2 and 3 standard errors (Figure 3). Then find how many lakes would give an interval at about 95\% that is as narrow as the 1-standard-error interval, if \hat{p} stays near 0.24.

Figure 3One sample, three levels (made-up data). Each step up in confidence widens the interval by another standard error on each side.

Solution

The width is twice the margin, so with \text{SE} \approx 0.0270 the widths are 2 \times 0.0270 = 0.054 (about 68\%), 0.108 (about 95\%) and 0.162 (about 99.7\%). More confidence from the same data costs width: the interval says more surely that it captures p by saying less precisely where p is. A 100\% interval would run from 0 to 1 and say nothing.

To keep about 95\% but cut the width to 0.054, the margin of 2\,\text{SE} must halve, so \text{SE} must halve. The standard error shrinks like \displaystyle \frac{1}{\sqrt{n}}, so that takes 4 times the sample, 4 \times 250 = 1{,}000 lakes. Checking,

\begin{aligned} 2\sqrt{\frac{0.1824}{1000}} &= 2\sqrt{0.0001824} \\ &\approx 2 \times 0.0135 = 0.027 \end{aligned}

giving 0.213 to 0.267. The sample of 1{,}000 is still under a tenth of 12{,}000, so the method’s conditions hold. Exact multipliers for levels such as 90\% and 99\%, where a 99\% interval needs a multiplier larger than 2, come from the Critical value.

Boundaries

The level describes the method over all possible samples, never one computed interval. A computed interval contains p or it does not, with no probability left in that statement.
An interval estimates a parameter. It does not describe the spread of individual values in the population, and it is not a range for the next sample’s statistic.
The level holds only for a random sample, analyzed by the stated method. A convenience or voluntary sample, nonresponse, or biased wording can push every interval the same way, and the stated 95\% then means nothing; no level corrects bias.
The level 95\% for \hat{p} \pm 2\,\text{SE} relies on a sampling distribution close to normal: enough successes and failures in the sample, and a sample under a tenth of the population (The ten percent condition). For small samples or proportions near 0 or 1 the true capture rate can fall well short of the stated level.
Judging a claimed value is one-sided in strength. A value outside the interval is implausible at that level; a value inside is only plausible, never confirmed.
A level is chosen before seeing the data. Picking the level after looking, to place a claim inside or outside, removes the meaning of the level.

Where this goes next

The rounded multiplier 2 gives way to the Critical value, the exact multiplier for any chosen level, and with it the full One sample interval for a proportion and sample size and, for means, the One sample interval for a mean and sample size. Every interval in those lessons is read the same way as here: a level that belongs to the method, an interval that names plausible values of a parameter in a population. Judging a claimed value against an interval is the first form of a significance test, which Deciding and concluding from a p-value makes precise.

Exercises

Exercises 1 to 9 go with the first part of the lesson (Examples 1 to 3), Exercises 10 to 18 with the second (Examples 4 to 7), and the reasoning group draws on both.

Part 1: building intervals and reading the level

A neighboring state has 20{,}000 lakes. A random sample of 400 of them finds 72 with zebra mussels. Find \hat{p}, the standard error, and the interval \hat{p} \pm 2\,\text{SE}.
Siiri repeats her simulation with 200 random samples instead of 50. About how many of the 200 intervals should capture p, and about how many should miss?
In Figure 1, sample 36 had 81 lakes with mussels. Find its interval, decide whether it captures p = 0.25, and find how many standard deviations of the sampling distribution its \hat{p} lies from p.
Another of Siiri’s samples of 250 lakes from the same made-up state has 51 with mussels. Find its interval and decide whether it captures p = 0.25.
Ten agencies in different states each compute a 95\% interval from independent random samples. Find the probability that all ten capture their true share, and the expected number of misses.
Siiri redraws the 50 intervals of Figure 1 at 1 standard error, \hat{p} \pm 1\,\text{SE}. About how many should capture p?

Part 1: reasoning about the level

After computing 0.186 to 0.294, Kaisa wants to write that p lies in it with probability 0.95. Explain what the 0.95 does describe, and why it does not attach to her computed interval.
Every interval in Figure 1 comes from a sample of 250 lakes. Explain why their widths are not all equal.
On a new batch of 50 samples, Siiri gets 49 captures and Matti, on another batch, gets 45. Does either result show that the method’s level is not about 95\%? Explain.

Part 2: interpreting in context

Write a sentence interpreting the interval from Exercise 1 in context, naming the parameter, the population and the level.
Matti surveys a random sample of 500 lakeshore property owners in the agency’s state, and 315 support a rule requiring boats to be washed before entering a lake. Find the interval at about 95\% and say whether it supports the statement that a majority of all lakeshore owners favor the rule.
Express the interval from Exercise 1 as a number of the neighboring state’s 20{,}000 lakes.

Part 2: judging claimed values

Using the agency’s interval 0.186 to 0.294, decide whether each claimed share is plausible at about 95\%: 0.18, 0.25, 0.29.
A group claims that two-thirds of the lakeshore owners in Exercise 11 support the rule. Is that plausible at about 95\%? At 1 standard error, about 68\%?

Part 2: confidence and width

For the sample in Exercise 1, find the intervals at 1 and at 3 standard errors and the width of each.
How many lakes from the neighboring state would give an interval at about 95\% half as wide as the one in Exercise 1, if \hat{p} stays near 0.18? Check the ten percent condition.
From the same sample, one interval is computed at 90\% and another at 99\%. Without computing either, say which is wider and why.
A third state reports a 95\% interval of 0.27 to 0.35 for its share of lakes with mussels, built as \hat{p} \pm 2\,\text{SE}. Find \hat{p}, the margin of error, the standard error, and about how many lakes were sampled.

Reasoning

A news story says: “Between 18.6\% and 29.4\% of lakes have mussels, a range that is 95\% certain to be right.” Rewrite it as a statement the interval supports, and say what changed.
Explain why a 100\% confidence interval for the share of lakes with mussels would be useless.
The lake association says that because 0.20 is inside the 95\% interval, the share has been shown to be 20\%. Explain what the interval does show about 0.20.
Explain why a larger sample can narrow an interval without lowering its level, while raising the level from the same sample always widens it.
Suppose Kaisa had sampled 250 lakes at random, but only from lakes with a public boat ramp. Explain what population the interval then describes, and why its 95\% says nothing about all the state’s lakes.

Answers

\displaystyle \hat{p} = \frac{72}{400} = 0.18; \displaystyle \text{SE} = \sqrt{0.18 \times 0.82 / 400} = \sqrt{0.000369} \approx 0.0192; margin \approx 0.0384; interval 0.142 to 0.218. The sample is under a tenth of the lakes, 400 \le 2{,}000.
About 200 \times 0.95 = 190 capture and about 10 miss; the actual count varies from batch to batch.
\displaystyle \hat{p} = \frac{81}{250} = 0.324; \displaystyle \text{SE} = \sqrt{0.000876096} \approx 0.0296; interval 0.265 to 0.383, which misses, since its lower end is above 0.25. The distance is \displaystyle \frac{0.074}{0.0274} \approx 2.70 standard deviations.
\displaystyle \hat{p} = \frac{51}{250} = 0.204; \displaystyle \text{SE} = \sqrt{0.000649536} \approx 0.0255; interval 0.153 to 0.255, which captures 0.25, barely. Its center is \displaystyle \frac{0.046}{0.0274} \approx 1.68 standard deviations from p, within 2.
0.95^{10} \approx 0.599; the expected number of misses is 10 \times 0.05 = 0.5.
About 68\% of 50, so about 34 capture and 16 miss.
The 0.95 is the probability, before sampling, that the method’s interval will capture p: the share of all possible samples whose interval captures it. After sampling, the interval is two fixed numbers and p is fixed, so p is inside or not; she can say she is 95\% confident, meaning the method succeeds for about 95\% of samples.
The standard error \displaystyle \sqrt{\hat{p}(1 - \hat{p})/n} uses each sample’s own \hat{p}, which varies from sample to sample, so the margins differ slightly even with n fixed.
No. The count of captures in 50 intervals varies by chance around 47.5; 49 and 45 are both ordinary results. The level is the long-run share over all samples, which a single batch only approximates.
We are 95\% confident that the interval from 0.142 to 0.218 captures the true proportion of all the neighboring state’s lakes that have zebra mussels.
\displaystyle \hat{p} = \frac{315}{500} = 0.63; \displaystyle \text{SE} = \sqrt{0.2331/500} = \sqrt{0.0004662} \approx 0.0216; margin \approx 0.0432; interval 0.587 to 0.673. Every plausible value is above 0.5, so at about 95\% the data support a majority.
0.1416 \times 20{,}000 \approx 2{,}832 and 0.2184 \times 20{,}000 \approx 4{,}368: between about 2{,}830 and 4{,}370 lakes, from the unrounded endpoints.
0.18 is below 0.186, so it is not plausible; 0.25 and 0.29 are inside, so both are plausible.
Two-thirds is about 0.667, inside 0.587 to 0.673, so it is plausible at about 95\%. At 1 standard error the interval is 0.608 to 0.652, which leaves it out.
At 1 standard error: 0.161 to 0.199, width 0.038. At 3: margin \approx 0.0576, so 0.122 to 0.238, width 0.115.
Half the width needs half the standard error, so 4 \times 400 = 1{,}600 lakes: \displaystyle 2\sqrt{0.1476/1600} \approx 0.0192, half of 0.0384. The condition holds, 1{,}600 \le 2{,}000.
The 99\% interval is wider. Capturing p for more samples needs a larger multiplier on the same standard error, and 99\% needs a multiplier larger than 2, while 90\% needs one smaller.
\displaystyle \hat{p} = \frac{0.27 + 0.35}{2} = 0.31; margin 0.04; \text{SE} = 0.02; \displaystyle n = \frac{0.31 \times 0.69}{0.02^2} = \frac{0.2139}{0.0004} = 534.75, about 535 lakes.
“We are 95\% confident that between 18.6\% and 29.4\% of all the state’s lakes have zebra mussels.” The 95\% now describes the method that made the range, not a chance that this range is right, and the population is named.
Only the range 0 to 1 captures every possible share for every sample. It is certain and says nothing: an interval is useful because it is narrow enough to rule values out.
It shows 0.20 is plausible at about 95\%, consistent with the data. So are 0.19, 0.24, 0.29 and every other value inside; the interval cannot single out one of them.
The width is 2 \times (\text{multiplier}) \times \text{SE}. The level fixes the multiplier, and the sample size enters only through \text{SE}, which falls like \displaystyle \frac{1}{\sqrt{n}}. From one sample the \text{SE} is fixed, so a higher level, a larger multiplier, can only widen it.
The random sample represents lakes with a public boat ramp, so the interval estimates the share among those lakes only, and its 95\% is the capture rate for that share. Boat ramps bring boats, which may carry mussels, so the ramp lakes may differ from the rest, and no level measures how far.