Response bias and question wording
A survey can ask exactly the right people and still get the wrong answer. How can the way a question is asked push the answers away from the truth?
Where you are 3 in Bias
Why this exists
A survey has two jobs: ask the right people, and get true answers from them. The first job, choosing who is asked, is the subject of convenience and voluntary response samples, and the fix there is to let chance choose.
Chance cannot do the second job. Once a person is chosen, the answer still has to travel from what is true, through memory, feelings and a reading of the question, onto the form, and it can be bent on the way. A person may shade an embarrassing answer, give the answer the asker seems to hope for, or guess when they cannot remember. The question itself can lean on them too.
So the convenience-sample lesson is about who answers; this lesson is about how they answer.
Definition
Response bias is a lean in a survey’s results that comes from the answers themselves: the people asked give answers that differ from the truth, and the differences tend in one direction. That one direction is what makes it bias rather than plain error.
| Source | What can happen | Which way it pushes |
|---|---|---|
| A sensitive question | The true answer is embarrassing | toward the answer that looks better |
| Wanting to please the asker | People guess what the asker hopes for | toward the hoped-for answer |
| Poor memory | People cannot recall, so they guess | leans only if something tilts the guess |
| Leading or loaded wording | The question praises, blames or assumes | toward the side the words favor |
| A double question | Two questions get one answer | the answers cannot be read |
| The order of the options | Some places in a list may be chosen more | toward the option in that place |
A leading question signals which answer is expected (“Don’t you agree that ...?”). A loaded question carries words with strong feeling, or an assumption the person may not share (“Should the council waste money on ...?” assumes the money is wasted). Leading and loaded wording together are called question wording bias. A double question (or double-barrelled question) asks about two things but allows one answer.
A question is neutral when it names the topic without praising or blaming any answer, asks about one thing, offers balanced answers, and is asked so that no answer is more comfortable to give than another.
Intuition
Hamnet’s form tutor asks each student, out loud in front of the class, how many hours a day they spend on screens. Suppose we could also see each student’s real daily average, from the screen-time count on their phone. Figure 1 shows ten made-up students, one per row.
Students A and D said their real hours. Every other arrow points left, by one hour or two, and not one points right. Honest guesses would miss both ways; these all went one way, perhaps because a smaller number sounds better said aloud to a teacher.
The ten could have been chosen perfectly, or been the whole class. The figure is about what they said, not who they were.
Results
Response bias survives a good sample, and a bigger one. A random sample, or even a census, removes a lean that comes from who is asked, not a lean in what people say. More people asked the same way give more of the same shaded answers.
The direction comes from one question: which answer is easier to give? Answers can drift toward the one that looks better, the one the asker seems to want, the one the wording favors, or the one in a favored place on the list.
The cure is in the asking: neutral wording, one question at a time, balanced options in a varied order, a private way to answer, an asker with no stake in the answer, and questions people can answer from memory. Where the true value can be measured rather than asked, measure it.
Why it is true
Each answer is the true value minus a shift, the amount that person shaded it. Adding up,
In Figure 1 the real hours add to 50 and the shifts add to 12, so the hours said add to 50 - 12 = 38.
If every shift is zero or downward, the total of the shifts is at least zero, so the total said is never more than the total true, and equals it only if nobody shaded. Honest guesses that are sometimes high and sometimes low partly cancel in a total; one-way shifts have nothing to cancel against. A hundred students with the same habits as these ten give totals of 500 and 380: bigger totals, the same means, so the lean is still there.
Worked examples
Screen time said out loud
Use Figure 1. Find the mean of the real hours, the mean of the hours said, and the gap. Name the likely sources of the bias, its direction, and a better way to collect the data.
SolutionThe real hours add to 50, so their mean is \displaystyle \frac{50}{10} = 5 hours. The hours said add to 38, so their mean is \displaystyle \frac{38}{10} = 3.8 hours. The survey comes out 5 - 3.8 = 1.2 hours a day too low.
Two sources could be at work. A large number of hours can sound like a confession, so the question is mildly sensitive; and the asker is a teacher, whose hoped-for answer is easy to guess. Both make the smaller number easier to say, so both push the answers down.
An unsigned slip in a box takes away the audience. Better still, read the number rather than ask it: each student copies the weekly average from the phone’s screen-time count.
Two loaded questions about a bike track
A council asks residents one of two questions about a proposed bike track in the park:
(a) “Should the council waste money on a noisy bike track that will fill our quiet park with crowds?”
(b) “Do you support the exciting new bike track that will get young people active outdoors?”
Say which way each pushes, and write one neutral question to replace both.
SolutionQuestion (a) is loaded against the track: “waste” assumes the money is wasted, and “noisy”, “crowds” and “our quiet park” name only costs. It pushes toward no. Question (b) is loaded in favor: “exciting” praises the track and “get young people active” names only a benefit. It pushes toward yes. Neither result would say what residents think.
A neutral version names the plan, states the plain facts a resident needs, and offers balanced answers: “The council is considering building a bike track in the park. It would be paid for from this year’s parks budget and would take up part of the open grass. Do you support or oppose building it?”
A longer lunch and shorter breaks
Kezia’s school council asks a random sample of 200 students: “Do you want a longer lunch break and shorter morning breaks?” 140 say yes. The newsletter reports that “70\% of students want shorter morning breaks.” Is that supported? Rewrite the question.
SolutionThe arithmetic is right: \displaystyle \frac{140}{200} = 0.7 = 70\% answered yes.
But this is a double question. A student who wants a longer lunch but dislikes shorter morning breaks might say yes, caring more about lunch; another with the same views might say no. A yes does not show which half the student agreed with, so the 70\% supports neither change on its own. The report is not supported.
If the changes are separate, ask two questions: “Do you want a longer lunch break?” and “Do you want shorter morning breaks?” If they are one trade, say so: “A longer lunch would mean shorter morning breaks. Would you rather have that, or keep both breaks as they are?”
Swims in a year
Isadora asks a random sample of 30 adults, “How many times did you go swimming last year?” Most answers are round numbers such as 10, 20 and 50. She then asks a different random sample of 30 adults, “How many times did you go swimming in the last 4 weeks?” and the answers add to 45. Use the second survey to estimate the mean number of swims an adult takes in a year, and say why it may be more trustworthy than the first.
SolutionThe second survey’s mean is \displaystyle \frac{45}{30} = 1.5 swims in 4 weeks. A year of 52 weeks holds \displaystyle \frac{52}{4} = 13 blocks of 4 weeks, so the estimate for a year is
Round answers are the mark of guessing rather than counting. Memory alone does not say which way a guess goes, but if people like to seem active, that could tilt the guesses up. Four recent weeks can actually be counted; the price is that they may be unusual, since a summer month could give far more swims than a winter one.
The order of the names
Jethro’s council asks a random sample of 300 residents to choose a name for a new community garden from 6 names, printed in the same order on every form. “Kingfisher Garden”, printed first, gets 96 votes. A second council asks about a playground with 6 names, but prints 6 versions of the form, each name first on one version, and hands out 50 of each version at random. “Willow Corner” gets 18 votes from the 50 forms on which it came first and 40 votes from the 250 on which it did not. Find each percent, and say what the second survey suggests about the first.
Solution“Kingfisher Garden” got \displaystyle \frac{96}{300} = 0.32 = 32\% of the votes.
“Willow Corner” got \displaystyle \frac{18}{50} = 36\% where it came first and \displaystyle \frac{40}{250} = 16\% elsewhere. Overall it got 18 + 40 = 58 votes, and \displaystyle \frac{58}{300} is about 19\%.
The forms were handed out at random, so the two groups differ only in the order of the list, yet the name’s share more than doubled in first place. “Kingfisher Garden” was first on every form, so its 32\% mixes how much residents liked the name with any help from the top place, and the two cannot be separated. Turning the order round gives every name the top place equally often.
Boundaries
Response bias is about what is said, not who says it. A survey can suffer from both at once, and fixing one does nothing for the other. Groups that had no chance of being asked, or chosen people who never reply, are a third problem, undercoverage and nonresponse.
The direction is not always knowable. Poor memory leans only if something tilts the guesses, and a double question does not lean at all; its answers simply cannot be read. Then the honest conclusion is that the result is unreliable, not that it is too high or too low.
Neutral does not mean empty. A question may state facts that matter to the answer, as long as they are plain and do not favor a side.
Removing a reason to shade is not proof of truth. An unsigned form takes away the audience, but a person may still shade an answer for their own comfort. Measuring, where possible, is stronger evidence than any wording.
Where this goes next
A simple random sample deals with who is asked; this lesson deals with what they say, and a sound survey needs both. A survey’s percent or mean is a statistic estimating the population’s parameter. Chance lets the statistic fall on either side of the parameter, as sampling variability describes, while response bias moves it the same way every time.
In experiments, a person who knows which treatment they had may report feeling better because they expect to, one reason experiments use a control group and, where possible, keep people from knowing which group they are in.
Exercises
Naming the source
- Before checking her register, a coach asks each of her 20 players, face to face, “How many training sessions did you miss last month?” The answers add to 18; the register shows 40 missed. Find the gap and the percent of missed sessions owned up to, and name the likely source of the bias and its direction.
- A survey asks, “Don’t you agree that the park needs more benches?” Name the kind of wording and its direction.
- A survey asks, “Are the town’s buses clean and on time?” Explain why the answers cannot be read.
- A survey asks, “How many hours of sleep did you get on each night of last month?” Name the source of trouble, say whether its direction can be known, and suggest a better question.
Rewriting neutrally
- Rewrite neutrally, and say which way the original pushes: “Wouldn’t a four-day school week give tired students the rest they badly need?”
- A town asks a random half of 500 residents, “Should the town keep its beautiful old clock tower, or pull it down for yet more parking?” and the other half a neutral version. 205 of the first 250 and 145 of the second 250 choose keeping the tower. Find both percents and the gap, say which way the original pushes, and write a neutral version.
- A concert survey offers the answers “Excellent / Very good / Good / Fair”. Which way does this list push the result? Write a balanced list.
- A form says, “Most people in this town vote in local elections. Do you?” A random half of 600 residents get this form and the other half get only “Do you vote in local elections?”; 252 of the first 300 and 204 of the second 300 say yes. Find both percents and the gap, and explain how the first sentence could push the answers, even if it is true.
Working with numbers
- Use Figure 1. How many of the 10 students said fewer hours than their real hours, how many said exactly their real hours, and how many said more? What was the largest gap, and who had it?
- A worker in a water company’s uniform asks 400 households, chosen at random, “Do you always turn the tap off while you brush your teeth?” and 340 say yes. Later the same households fill in an unsigned form, and 260 say yes. Find both percents and the difference, and name the likely sources of bias at the door. Which result is more trustworthy, and why does even that one not prove the true percent?
- Leopold-Ray asks a random sample of 40 people how many times they went swimming in the last 4 weeks, and the answers add to 52. Find the mean for 4 weeks and estimate the mean for a year of 52 weeks. A survey asking about “last year” got a mean of 24. Give a reason the two might differ.
- A council prints a poll with 4 names in 4 versions, each name first on one version, and hands out 150 of each at random. “Heron Walk” gets 66 votes from the 150 forms on which it came first and 90 votes from the 450 on which it did not. Find its percent in each group and overall, and say what the two groups show.
Reasoning
- Guinevere says, “Our survey asked every student in the school, so it cannot be biased.” Explain why she is wrong, with an example of a question that could still give a biased result.
- Explain the difference between voluntary response bias and response bias, with an example of each.
- Two random-sample polls of the same size, drawn from the same town in the same week, find opposite results on the same topic. Explain how response bias could account for this, and what you would want to see before trusting either.
Answers
- The gap is 40 - 18 = 22 sessions, and the answers owned up to \displaystyle \frac{18}{40} = 45\% of the missed sessions. The question is sensitive and is asked by the person it is most awkward to tell, who plainly hopes for a small number, so the answers are pushed down.
- A leading question: “Don’t you agree” signals the expected answer. It pushes toward agreeing.
- It is a double question. A “no” could mean dirty, or late, or both, and a person who finds the buses clean but late has no right answer to give.
- Poor memory: nobody can recall each night of a month, so the answers are rough guesses, and nothing obvious tilts them one way, so the direction cannot be known. Better: “How many hours of sleep did you get last night?”
- It pushes toward yes: “Wouldn’t” leads, and “tired students” and “rest they badly need” assume the benefit. Neutral: “Would you support or oppose changing to a four-day school week?”
- Original: \displaystyle \frac{205}{250} = 82\%; neutral: \displaystyle \frac{145}{250} = 58\%; a gap of 24 percentage points. The halves were chosen at random, so the wording is the likely cause. The original pushes toward keeping the tower: “beautiful” praises it and “yet more parking” belittles the other choice. Neutral: “Should the town keep the old clock tower or replace it with parking?”
- Three answers praise and the lowest is “Fair”, so nobody can say the concert was poor; the result is pushed up. Balanced: “Very good / Good / Neither good nor poor / Poor / Very poor”.
- With the sentence: \displaystyle \frac{252}{300} = 84\%; without: \displaystyle \frac{204}{300} = 68\%; a gap of 16 percentage points. The halves were chosen at random, so the sentence is the likely cause. It tells people which answer is expected and makes “no” sound unusual, so some who do not vote may say they do; the sentence being true does not change that.
- 8 said fewer, 2 (A and D) said exactly their real hours, and 0 said more. The largest gap was 2 hours, by students E, G, I and J.
- At the door \displaystyle \frac{340}{400} = 85\%; on the form \displaystyle \frac{260}{400} = 65\%; a difference of 20 percentage points. At the door the uniform shows the hoped-for answer, and “yes” sounds responsible, so both push toward yes. The unsigned form is more trustworthy, since nobody is there to please, but people may still shade for their own comfort; only measuring could prove the true figure.
- The mean is \displaystyle \frac{52}{40} = 1.3 swims in 4 weeks, and \displaystyle \frac{52}{4} = 13, so the estimate is 1.3 \times 13 = 16.9 swims a year. A whole year’s swims have to be guessed, and a wish to seem active could tilt the guesses up; or the 4 weeks may have been unusual, such as a cold month.
- First: \displaystyle \frac{66}{150} = 44\%. Elsewhere: \displaystyle \frac{90}{450} = 20\%. Overall: \displaystyle \frac{156}{600} = 26\%. The forms were handed out at random, so the groups differ only in order, and the name did more than twice as well in first place: position mattered a lot.
- Asking everybody removes a lean from who is asked, not from how they answer. “How many times have you copied homework this term?”, asked by a teacher, could still be pushed down.
- Voluntary response bias is about who answers: people choose to reply, so strong feelings are over-counted, as in a poll anyone can text in to. Response bias is about how chosen people answer: a randomly chosen person asked a leading question can give a bent answer.
- The polls may have worded the question differently (one loaded for, one against), ordered the answers differently, or been asked by people seen to take a side; a random sample guards against none of these. You would want the exact wording, the answers offered and their order, and who asked.