Statistics

Undercoverage and nonresponse

A poll draws its people by chance, so every voter seems to have a fair shot. What about the voters who were never on the list it drew from, or who were chosen and never answered?

You need

Simple random samples and the sampling frameMath 7

Where you are 3 in Bias

Statistics7

Sampling and study design6

Bias3

Convenience and voluntary response samples Response bias and question wording

Undercoverage and nonresponse

Why this exists

A simple random sample takes the choice of who is asked away from the person collecting the data, and that removes one large source of bias. But the chance draw is made from a list, and it lands on people who must then answer. Two gaps are left that no amount of careful drawing can close: people who were never on the list, and people who were drawn and never replied. This lesson names both gaps, shows when they push a result one way, and shows how to put a number on how far the second gap could move it.

Definition

The population is the whole group a study wants to describe. The sampling frame is the list the sample is actually drawn from (recapped from Simple random samples and the sampling frame).

Undercoverage happens when some members of the population are missing from the frame, or are less likely than others to be reached by the method, so they have little or no chance of being in the sample.

Nonresponse happens when an individual chosen for the sample cannot be reached or will not take part. If n individuals are chosen and r of them respond, the response rate is

\frac{r}{n}.

Each becomes undercoverage bias or nonresponse bias when the people left out differ from the people counted in the thing being measured, so the result leans the same way every time the method is used.

Intuition

Seini runs a made-up phone poll for a county. She wants the percent of the county’s registered voters who support a ballot measure to pay for new bus routes. Her list is every landline number in the county’s voter file. She dials 1{,}000 numbers chosen at random from it, and 380 voters answer the survey.

Figure 1Each box sits inside the one before it. The poll’s answers come only from the shaded box (made-up data).

Read Figure 1 from the outside in. The population is every registered voter. The frame is smaller: a voter whose only phone is a cell phone is not on it and has chance 0. Inside the frame, chance picks the sample fairly. Then 620 of the chosen voters never answer, and only the shaded box is heard. Randomness protects the step from frame to sample; it protects neither the step from population to frame nor the step from sample to respondents.

A missing group is harmless if it would have answered like everyone else. It does damage only when it differs on the question: cell-only voters may ride buses more, and people who never pick up an unknown number may be busier or less interested in politics.

Results

The lean is the share missed times the difference. Let p be the population’s proportion, p_R the proportion among those reached and p_M the proportion among those missed, with a fraction m of the population missed. Then

p_R - p = m\,(p_R - p_M).

There is no bias if nobody is missed (m = 0) or if the missed answer like the reached (p_M = p_R). A large gap with no difference does no harm; a small gap with a big difference can.

Nonresponse puts bounds on the result. If n people are chosen, r respond and y of the respondents say yes, then the proportion of yes among all n chosen lies between

\frac{y}{n} \quad\text{and}\quad \frac{y + (n - r)}{n}.

The width of that range is \dfrac{n - r}{n}, the nonresponse rate.

A larger sample cures neither. Neither formula contains the sample size. More numbers from the same list reach the same kinds of voters, and more calls at the same response rate leave the same fraction silent.

Why it is true

Split the population of N into N_R reached and N_M missed, with m = \dfrac{N_M}{N}. The population’s yes count is the sum of the two parts’ counts, so p is a weighted average:

p = (1 - m)\,p_R + m\,p_M.

Subtract from p_R:

\begin{aligned} p_R - p &= p_R - (1 - m)p_R - m\,p_M \\ &= m\,p_R - m\,p_M = m\,(p_R - p_M). \end{aligned}

For the bounds, the yes count among the n chosen is y plus the yes count among the n - r nonrespondents, and that second count is at least 0 and at most n - r. Dividing by n gives the two ends, and their difference is \dfrac{n - r}{n}.

Worked examples

Example 1

Naming the gaps in Seini’s poll

Name the population, the frame, one group undercovered and the nonrespondents in Seini’s poll, and find the response rate.

Solution

The population is every registered voter in the county, since that is the group the result is meant to describe. The frame is the list of landline numbers in the voter file, since the 1{,}000 numbers were drawn from it.

Voters whose only phone is a cell phone are not on the list, so they are undercovered: no draw from this list can reach them. Voters with an unlisted landline are missed the same way.

The nonrespondents are the 1{,}000 - 380 = 620 chosen voters who did not answer: no one picked up, the line was busy, or they declined. The response rate is

\frac{380}{1000} = 0.38 = 38\%.

Example 2

How far could the silent voters move it?

Of the 380 respondents, 209 support the measure. Find the percent of respondents in support, and the lowest and highest percent possible for all 1{,}000 chosen voters.

Solution

Among respondents, \displaystyle \frac{209}{380} = 0.55 = 55\% support it.

The 620 nonrespondents’ views are unknown. If every one of them is against, the support count stays at 209:

\frac{209}{1000} = 0.209 = 20.9\%.

If every one is in favor, the count rises to 209 + 620 = 829:

\frac{829}{1000} = 0.829 = 82.9\%.

Figure 2The 620 silent voters could put the chosen 1,000 voters’ support anywhere in the shaded band.

So the support among the voters Seini chose lies somewhere from 20.9\% to 82.9\%, a range 62 percentage points wide, which is exactly the nonresponse rate. The range straddles 50\%: the data alone cannot say whether a majority of the chosen voters support the measure. The extremes are unlikely, but nothing in the data rules them out.

Example 3

The size of the undercoverage lean

Ngaire, a county analyst, suggests (made-up figures) that 30\% of the county’s voters are cell-only, that 55\% of the voters on the landline list support the measure, and that 75\% of cell-only voters do. Find the support among all voters and the bias of a poll that perfectly reflects the landline list.

Solution

The landline voters are a fraction 0.7 of the voters and the cell-only voters 0.3, so the county’s support is the weighted average

\begin{aligned} p &= 0.7(0.55) + 0.3(0.75) \\ &= 0.385 + 0.225 = 0.61 = 61\%. \end{aligned}

A poll that matches the landline list exactly reports 55\%, so its bias is 0.55 - 0.61 = -0.06, six percentage points too low. The formula agrees: m(p_R - p_M) = 0.3(0.55 - 0.75) = -0.06. The lean is downward because the missed group supports the measure more than the reached group does.

Example 4

Dialing five times as many numbers

Ngaire proposes dialing 5{,}000 numbers from the same list. Suppose the response rate stays at 38\%. How many respond, how wide is the nonresponse range, and what happens to the undercoverage bias of Example 3?

Solution

5000 \times 0.38 = 1900 respond, and 5000 - 1900 = 3100 do not. The range for the chosen voters is \displaystyle \frac{3100}{5000} = 0.62 wide, the same 62 percentage points as before, because its width is the nonresponse rate and that has not changed.

The cell-only voters are still off the list, so the bias 0.3(0.55 - 0.75) = -0.06 is unchanged. What the larger sample does shrink is the chance variation from one random draw to the next (sampling variability). The result clusters more tightly, around the same wrong value.

Example 5

Following up a random part of the nonrespondents

Ruslan picks 100 of the 620 nonrespondents at random and calls each one repeatedly, at different times of day, until he reaches them or gives up. He reaches 80, and 28 of them support the measure. Estimate the support among nonrespondents and among all 1{,}000 chosen voters, and say what this suggests about the respondents’ 55\%.

Solution

Among the reached nonrespondents, \displaystyle \frac{28}{80} = 0.35 = 35\% support it. Since Ruslan’s 100 were picked by chance from the 620, take 35\% as the estimate for all 620: 0.35 \times 620 = 217 supporters. With the 209 respondents in favor,

\begin{aligned} \frac{209 + 217}{1000} &= \frac{426}{1000} \\ &= 0.426 = 42.6\%. \end{aligned}

The hard-to-reach voters support the measure less than the respondents do, so nonresponse was pushing the poll up, from about 42.6\% to 55\%. A careful follow-up of a random part costs less than chasing all 620, and is worth more than dialing new numbers, which would only add more people who answer easily. The estimate is not perfect: 20 of Ruslan’s 100 were never reached, and they may differ again.

Example 6

Which questions does the gap threaten?

The landline list misses cell-only voters. For each question, say whether undercoverage is likely to bias the result: (a) the percent of voters who rent their home; (b) the percent of voters born in December; (c) the percent of voters who use the county’s online registration site.

Solution

The test is whether cell-only voters plausibly differ on that question. No data on these voters is given here, so each answer below is a reasoned guess to be checked, not a known fact.

(a) Possibly biased, low. If people who rent and move often are less inclined to set up a landline at each new home, renters would be over-represented among the missed.

(b) Likely little bias. There is no reason a birth month should be linked to the kind of phone a person keeps, so the missed voters should look like the reached ones. This still rests on an argument, not on the data, since the missed were never asked.

(c) Possibly biased, low. If voters who manage without a landline are more at home online, the poll would undercount users of the site.

The same frame can be fine for one question and badly biased for another. Undercoverage is a property of the frame and the question together.

Boundaries

Bias needs a difference. Undercoverage or nonresponse alone does not prove a result wrong; it makes the result unsupported until someone argues, or checks, that the missing people answer like the rest.

The bounds need a bounded answer. The all-yes, all-no range works because each answer lies between 0 and 1. A mean can be bounded the same way only when the variable has known limits: hours worked last week lie between 0 and 168, so setting every nonrespondent to 0, then to 168, gives a range, though usually one too wide to help. For a variable with no upper limit, such as yearly income, no finite range can be given.

These gaps are about who is heard, not what they say. A respondent who answers inaccurately, or meets a question worded to push one answer, is response bias, which can occur even in a census with no gaps at all.

Nonresponse is not voluntary response. In a voluntary response sample nobody is chosen and people pick themselves. In nonresponse, chance chose the person and the person did not answer. Both let the people themselves decide who is counted, which is why both can lean.

Repairs narrow the gaps without proving them closed. A better frame (adding cell numbers, or drawing from addresses rather than phones) shrinks undercoverage. Repeated calls at different times, a second way to reply such as mail, a shorter survey and a careful follow-up of a random part of the nonrespondents shrink nonresponse. What cannot be measured, the views of people still unreached, stays an assumption.

Where this goes next

Undercoverage, nonresponse and response bias are the three ways a survey built on chance can still miss its target, and judging a study means asking about all three. The same idea, that chance protects only the step it controls, returns in experiments: random assignment and random selection protect different claims, and a study needs the right one for the conclusion it wants to draw.

Exercises

Population, frame and the gaps

A made-up state’s wildlife office wants to know how many days the people who fish in the state spent fishing last year. It mails a survey to names drawn at random from its list of fishing license holders. In this state, nobody under 16 needs a license. Name the population, the frame and one undercovered group, and give a question for which that group would likely change the result.
In Seini’s poll, say whether each voter shows undercoverage, nonresponse, response bias, or none of these. (a) A voter whose only phone is a cell phone. (b) A voter whose number was dialed and who hung up. (c) A voter on the list whose number was not drawn. (d) A voter who answered and said yes to sound agreeable, though she is against the measure.
The wildlife office mails 1{,}600 surveys and 712 come back. Find the response rate.

How far nonresponse could move a result

Of the 712 returned surveys, 445 say the trout season should open earlier. Find the percent of returns in favor, and the lowest and highest percent possible among all 1{,}600 chosen license holders, each to one decimal place.
A survey chooses 500 people, 450 respond, and 270 of them say yes. Find the percent of respondents saying yes and the range for all 500 chosen. Compare its width with Example 2.
What response rate keeps the range for the chosen group at most 10 percentage points wide?

The size of the lean

A frame covers 80\% of a population. Among covered members 40\% say yes; among the rest 60\% do. (a) Find the population percent and the bias of a result that matches the covered members exactly. (b) Find the bias if the rest also say yes at 40\%.
In the state of Exercise 1, 1 in 5 of the people who fish is under 16. License holders fished a mean of 14 days last year and under-16s a mean of 9 days. Find the mean for everyone who fishes, and the bias of a survey that matches the license holders exactly.

Bigger samples and follow-ups

Seini dials 2{,}000 numbers instead. 760 answer and 494 of them support the measure. Find the percent of respondents in favor and the range for all 2{,}000 chosen. What did doubling change?
The wildlife office picks 60 of the 888 license holders who did not return the survey at random and phones them. It reaches 48, and 18 favor an earlier season. Estimate the percent in favor among nonrespondents and among all 1{,}600 chosen, to one decimal place. Which way was nonresponse pushing Exercise 4’s result?
Ngaire suggests replacing each voter who does not answer with a new random number from the list, and dialing until 1{,}000 voters have answered. Explain why this does not remove nonresponse bias.

Reasoning

Explain why drawing a simple random sample from the landline list, however carefully, cannot remove undercoverage of cell-only voters.
Ruslan says a 38\% response rate proves the poll is wrong. Using the formula p_R - p = m\,(p_R - p_M), explain what is right and what is wrong in his claim.
Show that the range of possible results for the chosen group always has width equal to the nonresponse rate, whatever y is.
Explain the difference between nonresponse in Seini’s poll and a voluntary response sample, and why each can lean.
Ruslan says the 380 who answered are a simple random sample of the voters on the landline list, because the 1{,}000 numbers were drawn at random. Explain what is wrong.

Answers

Population: everyone who fished in the state last year. Frame: the list of license holders. Undercovered: people under 16, who need no license and so are not on the list. Days spent fishing would likely change, since young people’s fishing depends on school and on adults taking them, so their days may differ from license holders’.
(a) Undercoverage: not on the frame. (b) Nonresponse: chosen but not heard. (c) None: being left out by chance is the fair part of the draw; it causes sampling variability, not bias. (d) Response bias: she was heard, but her answer is inaccurate.
\displaystyle \frac{712}{1600} = 0.445 = 44.5\%.
\displaystyle \frac{445}{712} = 0.625 = 62.5\%. With 1600 - 712 = 888 silent: lowest \displaystyle \frac{445}{1600} \approx 27.8\%, highest \displaystyle \frac{445 + 888}{1600}, which is \displaystyle \frac{1333}{1600} \approx 83.3\%.
\displaystyle \frac{270}{450} = 60\%. Lowest \displaystyle \frac{270}{500} = 54\%, highest \displaystyle \frac{320}{500} = 64\%. Width 10 points against 62 in Example 2: a high response rate leaves little room to move.
The width is the nonresponse rate, so it must be at most 10\%: a response rate of at least 90\%.
(a) 0.8(0.4) + 0.2(0.6) = 0.44 = 44\%; bias 0.40 - 0.44 = -0.04, four points low, matching 0.2(0.4 - 0.6) = -0.04. (b) 0.2(0.4 - 0.4) = 0: no bias, though a fifth is missed.
0.8(14) + 0.2(9) = 11.2 + 1.8 = 13 days. The survey gives 14, a bias of +1 day.
\displaystyle \frac{494}{760} = 65\%. With 2000 - 760 = 1240 silent: lowest \displaystyle \frac{494}{2000} = 24.7\%, highest \displaystyle \frac{494 + 1240}{2000} = \frac{1734}{2000} = 86.7\%. The width is still 62 points, because the response rate is still 38\%. Only the chance variation among respondents shrank.
\displaystyle \frac{18}{48} = 37.5\%; 0.375 \times 888 = 333, so \displaystyle \frac{445 + 333}{1600}, which is \displaystyle \frac{778}{1600} \approx 48.6\%. Nonresponse was pushing the result up, from about 48.6\% to 62.5\%.
The replacements are also people who answer, so the final 1{,}000 are all easy-to-reach voters. Those who do not answer are still missing in the same proportion of the people tried, and if they differ on the question the lean remains.
Chance only chooses among the entries on the list. A cell-only voter is not an entry, so has chance 0 under every draw; only changing the frame can reach that voter.
Right: m = 0.62 is large, so any difference between respondents and nonrespondents is multiplied by a lot and the poll is not trustworthy as it stands. Wrong: if p_M = p_R the bias is 0, so a low response rate leaves the result unsupported, not proven wrong.
The ends are \displaystyle \frac{y}{n} and \displaystyle \frac{y + n - r}{n}, and their difference is \displaystyle \frac{n - r}{n}, in which y cancels. That is the fraction chosen who did not respond.
In Seini’s poll chance chose each voter; the chosen voter then decided whether to answer. In a voluntary response sample no one is chosen; people decide to answer an open call. Both lean when the decision to answer is linked to the answer.
A random sample of the list was chosen, but who among them answered was decided by the voters, not by chance. The 380 are the chosen voters who answer phone surveys, which can differ from the list as a whole.