Statistics

Test statistic

A sample of 250 spa visits found 35.6% with a massage booked, against a claimed 30%. Is a gap of 5.6 percentage points big or small?

You need

Null and alternative hypothesesStatistics Standard errorStatistics

Unlocks

One sample test for a meanStatistics One sample test for a proportionStatistics

Where you are 7 in Hypothesis testing

Statistics7

Statistical inference8

Hypothesis testing7

Null and alternative hypotheses Randomization tests

Test statistic

Deciding and concluding from a p-value Type I and Type II errors ▾ Show all 7 (2 more)

Why this exists

A geothermal spa in Iceland has a booking system that reports 30\% of visitors book a massage (all numbers in this lesson are made-up data). Bjarki, who manages the treatment rooms, thinks the true share is higher. In the language of Null and alternative hypotheses, the claim and his suspicion are

H_0\colon p = 0.30 \qquad H_a\colon p > 0.30

where p is the proportion of all visits that include a massage. He takes a random sample of 250 visits from the year’s records and finds 89 with a massage:

\hat{p} = \frac{89}{250} = 0.356

That is 0.056 above the claim. Is 0.056 a lot? On its own the number cannot say. Sample proportions vary from sample to sample, so even if p really were 0.30 a sample would rarely land exactly on it. A gap means something only next to the typical size of that variation. The Standard error lesson measured such spread for an estimate; here the spread is measured in the world the null hypothesis describes, and the gap is counted in those units. That count is the test statistic.

Definition

A test statistic measures how far a sample’s statistic sits from the null value, in units of the statistic’s standard deviation computed as if the null hypothesis were true:

\text{test statistic} = \frac{\text{statistic} - \text{null value}}{\text{SD under } H_0}

Here “SD under H_0” is the standard deviation of the statistic’s sampling distribution, worked out as if H_0 were true. When that standard deviation depends on a parameter H_0 does not fix, as for a mean, a standard error stands in for it (see Results).

For one proportion with null value p_0, the statistic is \hat{p}, and under H_0 its sampling distribution has standard deviation \displaystyle \sigma_{\hat{p}} = \sqrt{p_0(1 - p_0)/n}. The test statistic is called z:

z = \frac{\hat{p} - p_0}{\sqrt{\dfrac{p_0(1 - p_0)}{n}}}

The denominator uses p_0, never \hat{p}. Many books call this denominator the standard error of the test; it is a standard deviation in the sense of the Standard error lesson, because it uses a value of the parameter (the claimed one) rather than an estimate from the sample.

A test statistic has no units. It is read like a z-score: the sign says which side of the null value the sample fell on, and the size says how many standard deviations away.

Intuition

Figure 1 draws the sampling distribution of the sample proportion for samples of 250 visits, assuming H_0 is true: centered at 0.30, with standard deviation about 0.0290.

Figure 1Where sample proportions from 250 visits would land if the booking system’s 30\% were right (made-up data). The z scale counts standard deviations from 0.30; Bjarki’s 0.356 sits at z \approx 1.93.

Bjarki’s 0.356 lies out on the right side of that curve. The z scale under it relabels the sample-proportion axis: 0 under the null value, 1 one standard deviation above it, and so on. A test statistic is nothing more than this relabeling. It turns “0.056 above 0.30” into “1.93 standard deviations above what H_0 expected”, and the second form means the same thing whatever the sample size, the null value or the units.

Why compute the spread from p_0? A test asks one question: if H_0 were true, how unusual would this sample be? Inside that “if”, the parameter is known, it is p_0, and so the spread of \hat{p} is known exactly. Nothing has to be estimated, and using \hat{p} would quietly mix in a world where H_0 is false.

Results

The general form carries over to every test in the course; only the statistic and its standard deviation change.

One proportion: statistic \hat{p}, null value p_0, denominator \displaystyle \sqrt{p_0(1 - p_0)/n}; the result is called z.
One mean: statistic \bar{x}, null value \mu_0, denominator \displaystyle s_x/\sqrt{n}; the result is called t.
A difference: statistic \hat{p}_1 - \hat{p}_2 or \bar{x}_1 - \bar{x}_2, null value usually 0, denominator the difference’s standard deviation or standard error under H_0; the result is a z or a t.

For a mean the population standard deviation is rarely known, so a standard error stands in, and the extra variation it brings makes the statistic follow a The t distribution and degrees of freedom rather than a normal one. Those tests have their own lessons.

When H_0\colon p = p_0 is true and the conditions hold (random sample, independence, and large counts checked with the null value: np_0 \ge 10 and n(1 - p_0) \ge 10), the statistic z has approximately the standard normal distribution: mean 0, standard deviation 1.

For fixed \hat{p} and p_0, z grows like \displaystyle \sqrt{n}: four times the sample size doubles z.

How surprising a given z is, as a probability, is the p-value; the next lesson builds it. For now, a large |z| is surprising if H_0 is true, since a standard normal value lands beyond 2 in size only about 5\% of the time.

Why it is true

Assume H_0, so p = p_0. By Sampling distribution of a sample proportion, \hat{p} then has mean p_0 and standard deviation \displaystyle \sigma_{\hat{p}} = \sqrt{p_0(1 - p_0)/n}, and its shape is close to normal when the large counts hold with p_0. Subtracting the constant p_0 shifts every value, so \hat{p} - p_0 has mean p_0 - p_0 = 0 and the same standard deviation. Dividing by the constant \sigma_{\hat{p}} divides the standard deviation by it:

\text{mean of } z = \frac{0}{\sigma_{\hat{p}}} = 0 \qquad \text{SD of } z = \frac{\sigma_{\hat{p}}}{\sigma_{\hat{p}}} = 1

Shifting and rescaling do not change shape, so z is close to standard normal. This holds only because \sigma_{\hat{p}} is a fixed number. Had \hat{p} been used in the denominator, the denominator would change from sample to sample and the argument would not go through as stated.

The \displaystyle \sqrt{n} growth comes from the denominator: replacing n by 4n divides \displaystyle \sqrt{p_0(1 - p_0)/n} by \displaystyle \sqrt{4} = 2, and the same gap divided by half as much is twice as large.

Worked examples

Example 1

Bjarki’s massage bookings

Find and interpret the test statistic for Bjarki’s sample: 89 massages in a random sample of 250 visits, testing H_0\colon p = 0.30 against H_a\colon p > 0.30.

Solution

First the conditions, since without them the reading of z as standard normal fails. The visits are a random sample. The spa logs far more than 10 \times 250 = 2{,}500 visits a year, so independence holds by the The ten percent condition. Large counts are checked with p_0, because the test reasons inside H_0: 250 \times 0.30 = 75 and 250 \times 0.70 = 175, both at least 10.

Under H_0 the standard deviation of \hat{p} is

\sigma_{\hat{p}} = \sqrt{\frac{0.30 \times 0.70}{250}} = \sqrt{0.00084} \approx 0.0290

and the test statistic is

z = \frac{0.356 - 0.30}{0.0290} = \frac{0.056}{0.0290} \approx 1.93

(computed with the unrounded 0.028983). Interpretation: Bjarki’s sample proportion of massage bookings is about 1.93 standard deviations above the 0.30 the booking system claims. The positive sign points the way H_a predicts.

Example 2

Why not the sample’s own standard error?

Thora, who runs the front desk, asks why Example 1 did not use the standard error \displaystyle \sqrt{\hat{p}(1 - \hat{p})/n} from the sample. Compute z that way and explain which is right for the test.

Solution

With \hat{p} = 0.356,

\sqrt{\frac{0.356 \times 0.644}{250}} = \sqrt{0.000917056} \approx 0.0303

and the ratio is \displaystyle \frac{0.056}{0.0303} \approx 1.85, a little smaller than 1.93. The two differ because the spread of \hat{p} depends on p, and 0.356 is a different p from 0.30. The test’s question is how far 0.356 is from 0.30 in a world where p = 0.30, and in that world the spread is 0.0290. So z \approx 1.93 is the test statistic. The standard error from \hat{p} is the right spread for a confidence interval, which has no claimed value to assume.

Example 3

A negative statistic

The booking system also says 45\% of visitors use the steam cave. Thora suspects it is wrong in one direction or the other, so H_a\colon p \neq 0.45. In a random sample of 180 visits, 72 used the steam cave. Find and interpret z.

Solution

Large counts with p_0: 180 \times 0.45 = 81 and 180 \times 0.55 = 99. The sample proportion is \displaystyle \hat{p} = \frac{72}{180} = 0.40. Under H_0,

\sigma_{\hat{p}} = \sqrt{\frac{0.45 \times 0.55}{180}} = \sqrt{0.001375} \approx 0.0371

z = \frac{0.40 - 0.45}{0.0371} \approx -1.35

The sample is about 1.35 standard deviations below the claimed 0.45. The minus sign says “below”; with a two-sided H_a, a value 1.35 above would count as the same amount of evidence. A distance of about one and a third standard deviations is an ordinary amount of sampling variation, so this sample gives little reason to doubt the claim.

Example 4

Same share, more visits

Hekla takes a larger random sample, 1{,}000 visits, and finds 356 massages: the same \hat{p} = 0.356 as Bjarki. Find z and compare.

Solution

Under H_0\colon p = 0.30,

\sigma_{\hat{p}} = \sqrt{\frac{0.21}{1000}} = \sqrt{0.00021} \approx 0.0145

z = \frac{0.056}{0.0145} \approx 3.86

The gap is the same 0.056, but four times the visits halves the standard deviation, so the gap is now about 3.86 standard deviations, twice 1.93. A share of 0.356 is a modest surprise from 250 visits if p = 0.30, and a very large one from 1{,}000: big samples rarely stray far, so the same gap says more.

Example 5

A smaller gap, a larger statistic

The system claims 5\% of visitors book the private lagoon, and Bjarki suspects more. In the same random sample of 250 visits, 20 booked it. Compare this gap with the massage gap.

Solution

Here \displaystyle \hat{p} = \frac{20}{250} = 0.08, a gap of only 0.03, about half the massage gap. Large counts: 250 \times 0.05 = 12.5 and 250 \times 0.95 = 237.5, both at least 10. Under H_0\colon p = 0.05,

\sigma_{\hat{p}} = \sqrt{\frac{0.05 \times 0.95}{250}} = \sqrt{0.00019} \approx 0.0138

z = \frac{0.08 - 0.05}{0.0138} \approx 2.18

The smaller gap is the larger statistic. Proportions near 0 vary less than proportions near the middle, because p_0(1 - p_0) is 0.0475 here against 0.21 for massages. This is why tests divide: a raw gap, like the observed difference used in re-randomization, has to be judged against its own spread each time, and dividing by that spread puts every test on one scale.

Example 6

How many massages would reach 2?

With 250 visits and p_0 = 0.30, what is the smallest number of massages that gives z \ge 2?

Solution

Set z \ge 2 and undo the division, which keeps the inequality because \sigma_{\hat{p}} is positive:

\begin{aligned} \hat{p} &\ge 0.30 + 2 \times 0.028983 \\ &= 0.357966 \end{aligned}

The count is 250\hat{p}, so it must be at least 250 \times 0.357966 \approx 89.49. Counts are whole numbers, so the smallest is 90. Check: \hat{p} = 0.36 gives \displaystyle z = \frac{0.06}{0.028983} \approx 2.07, while Bjarki’s 89 gave 1.93. One more massage in the sample would have pushed him past 2.

Boundaries

The statistic assumes the data come from a random sample or a randomized experiment. It measures chance variation only; a biased way of choosing visits moves \hat{p} by an amount z cannot show.
Reading z against the standard normal needs the large counts checked with p_0 and independence of the observations (Conditions for one proportion). When np_0 or n(1 - p_0) is under 10, z can still be computed, but its distribution is not close to normal, and exact binomial probabilities should be used instead.
At p_0 = 0 or p_0 = 1 the denominator is 0 and z is undefined.
A large |z| is evidence against H_0, not a measure of how big or important the difference is. With enough data a tiny gap gives a large z (Cautions about significance).
A small |z| does not show H_0 is true. It says the data are consistent with H_0, and with many nearby values of p too.

Where this goes next

The p-value turns z into a probability: the chance, if H_0 were true, of a statistic at least as far out as the one observed in the direction H_a names. That probability drives the decision in Deciding and concluding from a p-value, and the whole procedure for a proportion is assembled in One sample test for a proportion. The same fraction, with a standard error in the denominator, becomes the t statistic of One sample test for a mean. Not every test statistic has this shape: the Expected counts and the chi-square statistic adds up squared gaps over many cells, but it still measures distance from what the null hypothesis expected.

Exercises

Computing z

Test H_0\colon p = 0.30 against H_a\colon p > 0.30 for massage bookings with a random sample of 400 visits, 134 of which include a massage. Find and interpret z.
The system says 60\% of visitors arrive by tour bus; Hekla suspects fewer. In a random sample of 150 visits, 81 arrived by bus. Find z.
The system says 12\% of visitors are under 18; Thora tests H_a\colon p \neq 0.12. In a random sample of 300 visits, 45 were under 18. Find z.
Half the visitors are said to rent a towel. In a random sample of 64 visits, 40 rented one. Find z.

Reading z

A test with H_a\colon p < p_0 gives z = -2.4. Say in words what that number means.
Two tests of claims about the spa give z = 0.4 and z = 3.1, both with H_a\colon p > p_0. Which sample is more surprising if its null hypothesis is true, and why?
With n = 250 and p_0 = 0.30, what sample proportion gives z = -1? About how many massages is that?
Bjarki tests H_a\colon p > 0.30 with a new sample and gets z = -0.8. What does the sign say about his suspicion?

The null value in the denominator

For the data in Exercise 1, compute the ratio using the sample’s standard error \displaystyle \sqrt{\hat{p}(1 - \hat{p})/n}. Which number is the test statistic?
A test of H_0\colon p = 0.04 uses a random sample of 200 visits with 14 successes. Check the large counts condition for the test.
For samples of 250 visits, which null value p_0 gives the largest \sigma_{\hat{p}}, and what is it?

Sample size

Suppose the sample proportion in Exercise 1 had come from 1{,}600 visits instead of 400. Find z, and say how it compares with Exercise 1.
With 250 visits and p_0 = 0.30, what is the smallest number of massages that gives z \ge 2.5?
If the sample proportion of massages is 0.33 and p_0 = 0.30, how many visits are needed for z to reach 2?

The general form

The spa claims visitors soak for 45 minutes on average. A random sample of 36 visits has \bar{x} = 48.6 minutes and s_x = 12 minutes. Compute \displaystyle t = \frac{\bar{x} - 45}{s_x/\sqrt{n}}, the one-mean form from Results, and say what it counts. (Only the arithmetic is asked; how to read t comes in its own lesson.)

Reasoning

Explain why the standard deviation in z is computed from p_0 rather than from \hat{p}.
Bjarki says z = 1.93 means the massage share is 1.93\% higher than claimed. Explain what it does mean.
Explain why a test statistic has no units, and why that is useful.
Hekla’s sample gives z = 0. What was \hat{p}, and does this prove H_0?
Explain why doubling the gap \hat{p} - p_0 doubles z, but doubling n multiplies z by only about 1.41.
Show that, if H_0 is true, z has mean 0 and standard deviation 1.

Answers

\displaystyle \hat{p} = \frac{134}{400} = 0.335; \displaystyle \sigma_{\hat{p}} = \sqrt{0.21/400}, which is \displaystyle \sqrt{0.000525} \approx 0.0229; \displaystyle z = \frac{0.035}{0.0229} \approx 1.53. The sample share is about 1.53 standard deviations above the claimed 0.30.
\displaystyle \hat{p} = \frac{81}{150} = 0.54; \displaystyle \sigma_{\hat{p}} = \sqrt{0.60 \times 0.40 / 150}, which is \displaystyle \sqrt{0.0016} = 0.04; \displaystyle z = \frac{0.54 - 0.60}{0.04} = -1.5, a sample 1.5 standard deviations below the claim.
\displaystyle \hat{p} = \frac{45}{300} = 0.15; \displaystyle \sigma_{\hat{p}} = \sqrt{0.12 \times 0.88 / 300}, which is \displaystyle \sqrt{0.000352} \approx 0.0188; \displaystyle z = \frac{0.03}{0.0188} \approx 1.60. Large counts: 36 and 264.
\displaystyle \hat{p} = \frac{40}{64} = 0.625; \displaystyle \sigma_{\hat{p}} = \sqrt{0.25/64} = 0.0625; \displaystyle z = \frac{0.125}{0.0625} = 2.
The sample statistic is 2.4 standard deviations below the null value, the direction H_a predicts, computed as if H_0 were true.
The one with z = 3.1: it sits more than three standard deviations above its null value, while 0.4 is well within ordinary sampling variation. Because both are in standard deviations, they can be compared even though the claims differ.
\hat{p} = 0.30 - 0.028983 \approx 0.271; 250 \times 0.271017 \approx 67.75, so about 68 massages.
The sample proportion fell below 0.30, the opposite of what H_a predicts. Such a sample gives no evidence for p > 0.30.
\displaystyle \text{SE} = \sqrt{0.335 \times 0.665 / 400}, which is \displaystyle \sqrt{0.0005569375} \approx 0.0236, giving \displaystyle \frac{0.035}{0.0236} \approx 1.48. The test statistic is 1.53 from Exercise 1, which uses p_0.
np_0 = 200 \times 0.04 = 8, under 10, so the condition fails, even though the 14 observed successes would pass. A test checks with p_0.
p_0 = 0.5, where p_0(1 - p_0) is largest: \displaystyle \sqrt{0.25/250} = \sqrt{0.001} \approx 0.0316.
\displaystyle \sigma_{\hat{p}} = \sqrt{0.21/1600} \approx 0.0115; \displaystyle z = \frac{0.035}{0.0115} \approx 3.06, twice 1.53, since \displaystyle \sqrt{1600/400} = 2.
\hat{p} \ge 0.30 + 2.5 \times 0.028983 = 0.372457, a count of at least 93.11, so 94 (z \approx 2.62; 93 gives z \approx 2.48).
\displaystyle \frac{0.03}{\sqrt{0.21/n}} = 2 gives \displaystyle \frac{0.21}{n} = 0.000225, so n \approx 933.3: 934 visits.
\displaystyle t = \frac{48.6 - 45}{12/\sqrt{36}} = \frac{3.6}{2} = 1.8. The sample mean is 1.8 standard errors above the claimed 45 minutes; its distribution is a t distribution, met later.
The test asks how unusual the sample would be if H_0 were true. Under H_0, p = p_0, so the spread of \hat{p} is exactly \displaystyle \sqrt{p_0(1 - p_0)/n}; nothing needs estimating, and \hat{p} would describe a different world.
It is a count of standard deviations, not a percentage: 0.356 is 5.6 percentage points above 0.30, and that gap is about 1.93 times the typical sample-to-sample spread of \hat{p} when p = 0.30.
The numerator and denominator are in the same units (a proportion, or minutes), so they cancel. That puts tests with different units, sample sizes and null values on one scale, read against the standard normal.
\hat{p} = p_0 exactly. It does not prove H_0: values of p near p_0 would also produce such a sample. It only means the data give no evidence against H_0.
z is the gap divided by \displaystyle \sqrt{p_0(1 - p_0)/n}. Doubling the gap doubles the numerator. Doubling n divides the denominator by \displaystyle \sqrt{2} \approx 1.41, so z grows by that factor.
Under H_0, \hat{p} has mean p_0 and standard deviation \sigma_{\hat{p}}. Subtracting p_0 gives mean 0 and leaves the spread; dividing by the fixed number \sigma_{\hat{p}} leaves mean 0 and makes the standard deviation \sigma_{\hat{p}}/\sigma_{\hat{p}} = 1.