Choosing the right procedure
A city department sends a question and a description of how its data were gathered. Before any arithmetic, which of the inference procedures fits, and how does the question itself say so?
Where you are 8 in Statistical inference
When you reach for it
The research office of the city of Barcelona (a made-up office) receives study requests from city departments: parks, transit, libraries, markets. Laia, Arnau and Mireia read each request and the plan for gathering the data, and before anyone computes anything they must decide which procedure answers it. Each procedure has its own lesson; this lesson is about the choice that comes first. Each statistic estimates one parameter under one kind of design, so a correct calculation with the wrong procedure answers a question nobody asked.
The procedure
Day 1: the variable, the groups, estimate or claim
Name the parameter first. Read the request for its population, its variable and the summary it wants. A categorical variable (satisfied or not, which metro line) is summarized by a proportion p, or by a whole distribution when there are more than two categories. A quantitative variable (a measured number with units) is summarized by a mean \mu. The question decides, not the raw data: a visit length in minutes is quantitative, but “the share of visits over an hour” is a proportion.
Count the groups. One population or process: a one-sample procedure. Two populations, or two treatments formed by random assignment: a two-sample procedure, whose parameter is a difference, p_1 - p_2 or \mu_1 - \mu_2. A categorical variable with three or more categories, or compared across three or more groups, calls for chi-square: goodness of fit for one group against a claimed distribution, homogeneity for several groups.
Estimate or claim. “How large”, “how much” or “what share” asks for an interval. “Is there evidence that”, “has it changed” or “is it more than” a stated value asks for a test. Many requests want both: the test says whether, the interval says how much. The chi-square tests have no interval in this course, because their parameter is a whole distribution.
The design sets the scope, not the formula. Random sampling lets a conclusion reach the population sampled; random assignment lets it speak of cause (Observational studies and experiments).
Day 2: relationships and pairs
Two variables on each individual. Two categorical variables on one sample call for the Tests for independence and homogeneity. Two quantitative variables call for regression: the parameter is the slope \beta of the population line, and the procedure is a t interval or test for the slope. A categorical explanatory variable with two values and a quantitative response is a comparison of two means, which is Day 1 again.
Paired or two-sample. When each individual is measured twice, or individuals are matched in pairs by the design, the data are Paired data: interval and test for a mean difference: the parameter is \mu_d, the mean of the differences, and the procedure is a one-sample t on the differences. When nothing links particular members of the two groups, the procedure is two-sample. The test is whether the design says which value goes with which. Pairing takes the variation between individuals out of the differences; treating paired data as two samples counts it as noise.
The condition most at risk. One condition is usually the shaky one: a small sample of a skewed variable, a rare outcome, a sample not quite random. Naming it is part of the choice.
The tables below gather the course’s procedures. Every z and t statistic has the same shape,
and every interval is
with critical value z^* or t^*.
| Variable and design | Parameter | Procedure |
|---|---|---|
| categorical, two values, one group | p | z interval or z test for a proportion |
| categorical, two values, two groups | p_1 - p_2 | z interval or z test for a difference |
| categorical, one group, claimed distribution | a distribution | Goodness of fit test |
| categorical, several groups | several distributions | chi-square test for homogeneity |
| two categorical, one sample | an association | chi-square test for independence |
| quantitative, one group | \mu | t interval or t test for a mean |
| quantitative, paired | \mu_d | paired t interval or test |
| quantitative, two independent groups | \mu_1 - \mu_2 | t interval or t test for a difference |
| two quantitative | \beta | t interval or t test for a slope |
| Estimate | Standard error | df |
|---|---|---|
| \hat{p}, interval | \displaystyle \sqrt{\tfrac{\hat{p}(1 - \hat{p})}{n}} | none |
| \hat{p}, test | \displaystyle \sqrt{\tfrac{p_0(1 - p_0)}{n}} | none |
| \bar{x} | \displaystyle \tfrac{s}{\sqrt{n}} | n - 1 |
| \bar{x}_d, paired | \displaystyle \tfrac{s_d}{\sqrt{n}} | n - 1 |
| \bar{x}_1 - \bar{x}_2 | \displaystyle \sqrt{\tfrac{s_1^2}{n_1} + \tfrac{s_2^2}{n_2}} | technology |
| b, slope | SE_b | n - 2 |
For \hat{p}_1 - \hat{p}_2 there is no df; the interval and the test use
with \hat{p}_c the pooled proportion. The chi-square tests use \displaystyle \chi^2 = \sum \frac{(O - E)^2}{E}, with \text{df} = k - 1 for goodness of fit with k categories and \text{df} = (r - 1)(c - 1) for a two-way table.
| Procedure | Conditions beyond random data and the 10\% condition |
|---|---|
| z for proportions | at least 10 successes and 10 failures in each group (using p_0, or the pooled \hat{p}_c, in a test) |
| t for means | each sample (for paired data, the differences) has n \ge 30 or shows no strong skew or outliers |
| chi-square | every expected count at least 5 |
| t for a slope | a linear relationship, residuals with no strong skew or outliers, and about equal spread along the line |
Worked examples
Day 1: three requests, three procedures
For each request (made-up), name the parameter, the procedure, the reason, and the condition most at risk.
(a) Libraries: “What share of card holders used a branch in the last month?” Laia will ask a random sample of card holders.
(b) Transit: “The night bus schedule promises a bus every 20 minutes. Is the mean wait longer than that?” Arnau will time waits at randomly chosen stops and times.
(c) Markets: “How much longer do shoppers stay at Mercat de Sant Antoni than at Mercat de la Concepció?” Mireia will sample shoppers at each market separately.
Solution(a) Each card holder used a branch or did not: categorical, two values, one population, and “what share” asks for an estimate. The parameter is p, the proportion of all card holders who used a branch last month: a one-sample z interval. At risk is the random condition: card holders who do not answer may differ from those who do.
(b) A wait is quantitative, in one population of waits, and “longer than 20” is a claim about a stated value: a one-sample t test of H_0\colon \mu = 20 against H_a\colon \mu > 20, where \mu is the mean wait in minutes. At risk is shape. A wait cannot fall below 0 but can run long, so waits are likely skewed right, which matters if fewer than 30 are timed.
(c) Quantitative, two separate samples, and “how much longer” asks for an estimate: a two-sample t interval for \mu_1 - \mu_2, the difference in mean visit length. At risk: shoppers approached at one hour represent only that hour.
Day 1: the same data, two parameters
Laia records the length, in minutes, of each visit in a random sample of visits to the central library. Two requests arrive: (a) “Is the mean visit longer than an hour?” (b) “Do more than a third of visits last over an hour?” Choose a procedure for each.
Solution(a) A mean of a quantitative variable against a stated value: a one-sample t test of H_0\colon \mu = 60 against H_a\colon \mu > 60, with \mu the mean visit length in minutes.
(b) Each visit is reduced to yes or no, over an hour or not, so the variable the question uses is categorical: a one-sample z test of \displaystyle H_0\colon p = \frac{1}{3} against \displaystyle H_a\colon p > \frac{1}{3}, with p the proportion of visits over an hour.
Recording a number does not commit the office to a mean. The tests can disagree, since a few very long visits can lift the mean above 60 while most visits are short. At risk: skew for (a); for (b), large counts, since np_0 \ge 10 with \displaystyle p_0 = \frac{1}{3} needs at least 30 visits.
Day 1: two metro lines, a computed test
The transit department asks whether riders of metro lines L1 and L5 differ in the share who rate the service as good. Arnau takes separate random samples: of 250 L1 riders, 155 rate it good; of 200 L5 riders, 108 do (made-up data). Choose the procedure and carry it out at \alpha = 0.05.
SolutionChoosing. Good or not is categorical with two values; there are two independent samples; “do they differ” is a claim. The parameter is p_1 - p_2, the L1 minus the L5 proportion rating the service good, and the procedure is the two-sample z test of H_0\colon p_1 = p_2 against H_a\colon p_1 \ne p_2.
Conditions. Two separate random samples, each far less than 10\% of its line’s riders. In a test the large counts use the pooled proportion \displaystyle \hat{p}_c = \frac{263}{450} \approx 0.584: the counts 250(0.584) \approx 146.1, 250(0.416) \approx 103.9, 200(0.584) \approx 116.9 and 200(0.416) \approx 83.1 are all at least 10.
Statistic. With \displaystyle \hat{p}_1 = \frac{155}{250} = 0.62 and \displaystyle \hat{p}_2 = \frac{108}{200} = 0.54,
(the last value from unrounded inputs), and the two-sided P-value is about 0.087.
Conclusion. Since 0.087 > 0.05, fail to reject H_0: no convincing evidence that the share rating the service good differs between L1 and L5.
Day 1: more than two of anything
Choose the procedure and give the df (made-up designs).
(a) Mireia takes separate random samples of shoppers at four markets and records whether each paid by card. “Does the share paying by card differ among the four markets?”
(b) Arnau takes a random sample of bike-share trips and records the day of the week each began. The department claims trips are spread evenly over the seven days.
Solution(a) A yes or no variable in four groups. Running a two-sample z test on all 6 pairs at \alpha = 0.05 each raises the chance of some false alarm well above 0.05. One chi-square test for homogeneity tests all four at once, with H_0 that the card share is the same at all four markets. The table is 4 by 2, so \text{df} = (4 - 1)(2 - 1) = 3. At risk: expected counts, if card payment is rare at some market.
(b) One categorical variable with seven categories, one sample, and a claimed distribution of \displaystyle \frac{1}{7} per day: the chi-square goodness of fit test, \text{df} = 7 - 1 = 6. Each expected count is \displaystyle \frac{n}{7}, at least 5 once n \ge 35.
Day 2: two variables on each individual
Choose the procedure (made-up designs).
(a) Parks: Laia’s random sample of residents records each one’s district (ten districts) and main park activity (walking, sports, children’s play, sitting). “Is park activity related to district?”
(b) Transit: for a random sample of bus routes, Arnau records route length in kilometers and mean delay in minutes. “How much does delay grow per extra kilometer?”
(c) Libraries: Mireia’s random sample of residents records whether each holds a library card and how many books each read last year. “Do card holders read more?”
Solution(a) Two categorical variables on one sample: the chi-square test for independence, H_0 being that district and main activity are independent among residents. The table is 10 by 4, so \text{df} = (10 - 1)(4 - 1) = 27. At risk: 40 cells need a large sample for every expected count to reach 5.
(b) Two quantitative variables, and “how much per kilometer” asks for an estimate of a rate: a t interval for the slope \beta of the population line of mean delay on length. At risk is equal spread: longer routes may scatter more, a fan shape in the residual plot.
(c) One categorical variable with two values and one quantitative: a comparison of two means. The two groups are separate subsets of one random sample, so they act as independent samples: a two-sample t test of \mu_1 - \mu_2, card holders minus others, H_a\colon \mu_1 > \mu_2. At risk is shape, since book counts have a long right tail.
Day 2: paired or two-sample
Paired or two independent samples? Name the parameter.
(a) 30 volunteers each find a book with both versions of the library catalog website, in random order, and each search is timed.
(b) 60 volunteers are randomly split into two groups of 30, one for each version.
(c) In each of 12 parks, a coin decides which of the north and south lawns gets new grass seed; the other gets the old seed.
(d) Arnau measures noise at the 15 stops of one bus line and the 15 stops of another. Both lines number their stops 1 to 15.
Solution(a) Paired: each volunteer gives both times. The parameter is \mu_d, the mean of (old version time minus new version time).
(b) Two samples, since no volunteer is linked to a particular volunteer in the other group: \mu_1 - \mu_2.
(c) Paired by matching: the two lawns of a park share soil and weather, and the design pairs them before treatment. The parameter is the mean of (new seed height minus old seed height). The coin assigned the seed, so a rejection would be evidence that the seed causes the difference.
(d) Two samples: the shared numbering is a label, not a link made by the design.
Day 2: one study from start to finish
The transit department writes: “Last spring we gave buses priority at traffic signals. Did trip times fall, and by how much?” Arnau randomly selected 10 of the department’s 120 bus routes and found each route’s mean weekday midday trip time, in minutes, in the month before and the month after the change (made-up data). Choose the procedure, test at \alpha = 0.05, give a 95\% interval, and say what the result shows.
| route | before | after | difference |
|---|---|---|---|
| 1 | 41.2 | 38.1 | 3.1 |
| 2 | 35.6 | 33.8 | 1.8 |
| 3 | 48.9 | 46.5 | 2.4 |
| 4 | 29.4 | 29.9 | -0.5 |
| 5 | 52.3 | 48.3 | 4.0 |
| 6 | 33.0 | 31.8 | 1.2 |
| 7 | 45.7 | 42.8 | 2.9 |
| 8 | 27.8 | 27.2 | 0.6 |
| 9 | 38.1 | 36.6 | 1.5 |
| 10 | 44.6 | 42.6 | 2.0 |
Choosing. Trip time is quantitative, and each route was measured twice, so the data are paired. The parameter is \mu_d, the mean of (before minus after) over all 120 routes. “Did they fall” asks for a test and “by how much” for an interval: the paired t test and interval, with \text{df} = 10 - 1 = 9. H_0\colon \mu_d = 0; H_a\colon \mu_d > 0, since a fall makes before minus after positive.
Conditions. The routes are a random sample, and 10 is at most 10\% of 120. With only 10 pairs, the condition most at risk is shape: the differences must show no strong skew or outliers. The dot plot shows one negative value, an even spread and no outlier.
Statistic. The differences have \bar{x}_d = 1.9 minutes and s_d \approx 1.300 minutes, so
with 9 df and a one-sided P-value of about 0.0006.
Conclusion. Since 0.0006 \le 0.05, reject H_0: there is convincing evidence that mean midday trip time on the department’s routes was lower in the month after the change than in the month before.
Interval. With t^* \approx 2.262 for 9 df,
so the department can be 95\% confident that mean trip time fell by between about 0.97 and 2.83 minutes.
Had the routes been treated as two samples. Routes differ greatly in length (standard deviations about 8.284 minutes before and 7.167 after), so the two-sample standard error would be \displaystyle \sqrt{\frac{8.284^2}{10} + \frac{7.167^2}{10}} \approx 3.464, not 0.411, and t only about 0.549. Pairing lets the steady drop on each route show through.
Scope. The random sample lets the conclusion reach all 120 routes. But nothing was randomly assigned, and anything else that changed between the months, such as road work, could also have shortened trips: the data show that times fell, not that the signals caused it. Whether about 2 minutes on a trip averaging about 40 (roughly 5\%) is worth the cost is a question of practical significance, which the P-value does not answer.
Boundaries
- Three or more means. No procedure here compares the means of three or more groups; that needs analysis of variance, beyond this course. Running every two-sample t test in turn raises the chance of a false alarm.
- Random data. Every procedure assumes random sampling or random assignment. The right formula does not rescue a convenience sample.
- Failed conditions. If the condition at risk fails, the procedure does not apply: collect more data, combine categories for a reason fixed in advance, or use a method beyond this course.
- Chi-square has no direction and no interval here. For a direction, or “how much” with two groups and a yes or no variable, use the two-sample z procedures.
- Cause comes from the design, never from the procedure.
Where this goes next
This lesson closes the course. Beyond it lie analysis of variance for several means, multiple regression for several explanatory variables, and resampling methods that drop the shape conditions.
Exercises
All designs and data are made up. Unless a computation is asked for, name the parameter in context, the procedure, the reason, and the condition most at risk. Give P-values from technology.
Day 1: one variable, one or two groups
- Libraries: Laia asks a random sample of 300 card holders whether each borrowed an e-book this year. The department wants to know what share did.
- Transit: on a random sample of 40 weekday mornings, Arnau records how many minutes late the first train on metro line L3 runs. “Is it more than 2 minutes late on average?”
- In a random sample of 225 dog owners in the city, 81 say they use a dog park every week. Give a 95\% confidence interval for the proportion of all the city’s dog owners who do.
- In a random sample of 16 night bus trips, the delay at the last stop has mean 4.6 minutes and standard deviation 2.4 minutes, with no strong skew. Test at \alpha = 0.05 whether the mean delay is more than 3 minutes.
- Mireia plans to ask a random sample of 60 market vendors whether they sell only organic produce, and to test H_0\colon p = 0.10. Which condition fails, and why?
- A random sample of library visits records how many books were borrowed on each. (a) “Is the mean more than 2 books per visit?” (b) “Do fewer than a quarter of visits end with no book borrowed?”
- A random sample of bike-share trips records which of four kinds of station each began at (beach, center, hill, outer). The department’s plan says 20\%, 40\%, 15\% and 25\%. “Do trips match the plan?” Give the df.
- Separate random samples of visitors to five parks record whether each was satisfied. “Does the share satisfied differ among the parks?” Give the df.
Day 2: relationships, pairs and scope
- A random sample of 500 residents records each one’s age group (four groups) and preferred book format (print, e-book, audio). “Is preferred format related to age?” Give the df.
- For 25 randomly chosen parks, Laia records area in hectares and weekend visitors. “About how many more visitors does each extra hectare bring?” Give the df and t^* for 95\% confidence.
- “Is the mean commute shorter with the new trip-planning app?” (a) 18 randomly chosen riders each time the same commute with the old app one day and the new app another, the order set by a coin. (b) 40 volunteers are randomly split into two groups of 20, one per app.
- For a random sample of 18 bus routes, the least-squares line of mean delay (minutes) on number of stops has slope b = 0.42 with SE_b = 0.15, and the conditions hold. Test at \alpha = 0.05 whether delay is linearly related to the number of stops.
- At 12 randomly chosen library branches, Mireia records daily visitors under the new opening hours minus daily visitors under the old. The differences have mean 14.5 and standard deviation 9.2, with no strong skew. Give a 95\% confidence interval for the mean difference.
- Parks: 20 pairs of flowerbeds are matched by sunlight, and in each pair a coin decides which bed gets a new mulch. Name the parameter and procedure, and say whether a rejection could show that the mulch causes taller plants.
- Separate random samples of 200 riders on L2 and 200 on L4 record whether each felt crowded. Before seeing the data, the department asked whether L4 riders feel crowded more often. Why does the two-sample z test fit better than a chi-square test on the 2 by 2 table?
- Describe a design for Example 7 that would let Arnau conclude that the priority signals caused shorter trips, and name the procedure it calls for.
Reasoning
- Explain why the paired t procedure is a one-sample t procedure, and on what.
- A department asks “how much higher” a share is at one market than at another. Explain why a chi-square test cannot answer this as asked, and what can.
- A department asks whether mean waits differ among five metro lines. Explain why no procedure in this course fits, and what a two-sample t test on every pair of lines would risk.
- Explain why random sampling and random assignment leave a procedure’s computation unchanged but change what its conclusion may say.
Answers
- p, the proportion of all card holders who borrowed an e-book this year: one-sample z interval (yes or no, one population, “what share”). At risk: nonresponse spoiling the random condition.
- \mu, the mean minutes late of the first L3 train on weekday mornings: one-sample t test, H_0\colon \mu = 2, H_a\colon \mu > 2. At risk: independence, since nearby mornings can share one fault.
- From about 0.297 to 0.423. \displaystyle \hat{p} = \frac{81}{225} = 0.36, \displaystyle SE = \sqrt{\frac{0.36(0.64)}{225}} = 0.032, so 0.36 \pm 1.96(0.032) = 0.36 \pm 0.0627. The counts 81 and 144 are at least 10.
- \displaystyle t = \frac{4.6 - 3}{2.4 / \sqrt{16}} = \frac{1.6}{0.6} \approx 2.667, \text{df} = 15, P-value about 0.0088. Reject: convincing evidence that the mean delay at the last stop is more than 3 minutes.
- Large counts: np_0 = 60(0.10) = 6, below 10. A sample of at least 100 vendors would meet it.
- (a) \mu, mean books per visit: one-sample t test, H_a\colon \mu > 2; at risk is skew. (b) p, the proportion of visits with no book: one-sample z test, H_a\colon p < 0.25; large counts need n(0.25) \ge 10, so at least 40 visits.
- Chi-square goodness of fit (one categorical variable, a claimed distribution), \text{df} = 4 - 1 = 3. At risk: the hill expected count 0.15n must reach 5, so n \ge 34.
- Chi-square test for homogeneity, a 5 by 2 table: \text{df} = (5 - 1)(2 - 1) = 4. At risk: expected counts, if nearly everyone is satisfied at some park.
- Chi-square test for independence (two categorical variables, one sample): \text{df} = (4 - 1)(3 - 1) = 6. At risk: expected counts in rare cells such as audio among the oldest.
- t interval for the slope \beta of the population line of weekend visitors on area: \text{df} = 25 - 2 = 23, t^* \approx 2.069. At risk: one very large park acting as an influential point.
- (a) Paired: \mu_d, the mean of (old app time minus new app time), paired t test with H_a\colon \mu_d > 0 and \text{df} = 17; at risk is an outlier among the 18 differences. (b) Two independent groups by random assignment: two-sample t test of \mu_1 - \mu_2, old minus new; at risk is shape with 20 per group.
- \displaystyle t = \frac{0.42}{0.15} = 2.8, \text{df} = 16, two-sided P-value about 0.013. Reject H_0\colon \beta = 0: convincing evidence that mean delay is linearly related to number of stops, about 0.42 minutes more per stop in the sample.
- From about 8.65 to 20.35 visitors per day: t^* \approx 2.201 for 11 df, and \displaystyle 14.5 \pm 2.201 \cdot \frac{9.2}{\sqrt{12}} = 14.5 \pm 5.85.
- \mu_d, the mean of (mulch bed height minus other bed height): paired t test with \text{df} = 19. Yes: the coin assigned the mulch, so a rejection is evidence of cause, for beds like these.
- The request has a direction fixed in advance, H_a\colon p_{L4} > p_{L2}. The z test gives the one-sided P-value; chi-square squares away the direction.
- Over the same months, randomly assign half of a random sample of routes to priority signals and keep the old timing on the rest, recording each route’s change in trip time. A two-sample t test compares the mean changes; random assignment spreads road work and holidays over both groups alike.
- Each pair is reduced to one number, its difference, and the differences are one sample from a population with mean \mu_d. The standard error \displaystyle \frac{s_d}{\sqrt{n}} and \text{df} = n - 1 are those of the one-sample t procedure.
- A chi-square test only tests, its parameter is a whole distribution, and it gives no interval or direction here. “How much higher” asks for an estimate of p_1 - p_2, which the two-sample z interval gives.
- The variable is quantitative with five groups, and this course compares at most two means. Testing all 10 pairs at \alpha = 0.05 each makes some false alarm far more likely than 0.05 when no lines differ; analysis of variance tests all five at once.
- The computation uses only the numbers. What they stand for depends on how chance entered: random sampling makes the sample stand for a population; random assignment makes the groups alike apart from the treatment, so a difference can be credited to it.