Convenience and voluntary response samples
The people easiest to ask are right in front of you, and some people are keen to answer without being asked. Why can their answers tell you the wrong thing about everyone?
Where you are 3 in Bias
Why this exists
To learn about a large group you usually look at part of it, a sample from the population. Choosing that part well takes work, and two shortcuts save it: ask whoever is near you, or put out an open invitation and take whoever answers. Both are cheap, and both can be badly wrong, because the people near you, or the people who bother to answer, are often unlike the rest in the very thing you are asking about. This lesson names the two shortcuts and shows how to tell, from a study’s description, who was left out or counted too often, and which way that pushes the result.
Definition
A convenience sample is made of the individuals easiest for the person collecting the data to reach: those nearby, passing by, or already at hand. The collector decides who is in it, and decides by what is easy.
A voluntary response sample is made of the individuals who choose to answer an open invitation: they phone in, click, text, or post a form back. Nobody is picked; each respondent picks himself or herself, so it is also called a self-selected sample.
A sample represents its population when its mix matches the population’s mix in whatever matters for the question. A way of choosing is biased when it keeps favoring some parts of the population over others, so its results lean the same way time after time (the general idea is bias). The lean that comes from letting people choose whether to answer is called voluntary response bias.
| Method | Who decides who is in the sample | Who tends to be in it |
|---|---|---|
| Convenience | the collector, by what is easy | whoever is at that place at that time |
| Voluntary response | each respondent | whoever cares enough about the question to answer |
Intuition
Barnaby wants to know what the shoppers at a made-up mall think about longer opening hours, and asks everyone who walks in at the north entrance on weekday mornings. Figure 1 shows the mall’s 360 shoppers in one week. The filled dots, 180 of them, are shoppers who work on weekdays in the daytime.
Barnaby’s 24 shoppers all come from the shaded corner, and only 3 of them are filled. Someone at work from nine to five can only be at the mall on a weekday morning on a day off, so the corner is short of exactly those people. The rest of the grid was never going to be asked.
A voluntary response sample goes wrong another way. A made-up radio station plans to move its morning news from seven o’clock to eight, and invites listeners to phone in with their view. Figure 2 shows its 200 listeners, with the 50 who object filled.
Everyone heard the invitation, so nobody was out of reach. But objectors have a reason to pick up the phone, and listeners who do not mind mostly do not: 30 of the 50 objectors call, and 10 of the 150 others. A quarter of the listeners object; three quarters of the callers do. The opinion being measured is the same thing that decided who was measured.
Results
A convenience sample speaks only for the group it could reach. If that group differs from the rest of the population in the thing being measured, the result leans toward that group’s answer.
A voluntary response sample over-counts the people who feel most strongly. Each person’s chance of answering depends on how he or she feels about the question, so the result leans toward whichever side feels more strongly.
The direction of the lean comes from a comparison. Ask how the group reached, or that chose to answer, differs from the group missed, in the thing being measured. The result is pushed toward the reached group’s answer.
A bigger sample of the same kind does not remove the lean. More of the same people only settles the result more firmly on the reached group’s answer.
Why it is true
Split the population into the part the method reaches (or that chooses to reply) and the rest. Every member of the sample comes from the first part, so even a perfect small copy of that part shows that part’s percent.
The population’s percent is a blend of its two parts, and a blend sits between its parts. In Figure 2 the 40 callers are 75\% objectors, the other 160 listeners are \displaystyle \frac{20}{160} = 12.5\%, and all 200 together are 25\%, in between. So the sample lands on the reached part’s side of the population’s figure. Only when the two parts agree is there no lean, which is why a convenience sample is untrustworthy rather than always wrong.
Double everything, 400 listeners with the same habits, and \displaystyle \frac{60}{80} = 75\% of the callers still object. A bigger sample of the same kind is a bigger handful of the same part.
Worked examples
The mall entrance
The mall’s manager wants to know whether its shoppers would come more often if it stayed open until ten at night. Barnaby asks the shoppers at the north entrance on weekday mornings, as in Figure 1. Name the method, the group missed, and the way the result is likely pushed.
SolutionBarnaby takes the people who pass him at a time and place that suit him: a convenience sample.
The population is everyone who shops at the mall. Shoppers who work weekday daytime hours are mostly at work on weekday mornings, so they are largely missed; only 3 of the 24 asked are such shoppers, against half of the week’s shoppers.
Compare the two groups on the question. Late opening matters most to people who cannot shop during the working day, which is the missed group; weekday-morning shoppers can already come in daylight. The result is likely pushed down: the survey will show less interest in late opening than the shoppers as a whole feel.
The call-in poll
In Figure 2, 40 of the station’s 200 listeners phone in, and 30 of the callers object. Find the percent of callers and of all listeners who object, and explain the difference.
SolutionThe callers chose to answer an open invitation, so they are a voluntary response sample. Of them,
object, while of all the listeners \displaystyle \frac{50}{200} = \frac{1}{4} = 25\% object.
The two groups had very different chances of being in the sample. An objector called with chance \displaystyle \frac{30}{50} = \frac{3}{5}, a listener who did not mind with chance \displaystyle \frac{10}{150} = \frac{1}{15}, and \displaystyle \frac{3}{5} \div \frac{1}{15} = 9. An objector was nine times as likely to be in the sample, so objectors are over-counted and the result is pushed up, from 25\% to 75\%.
The “rate this article” stars
A made-up news website ends each article with “Rate this article” and five stars to click. An article on a change to the days the bins are collected was read 5{,}000 times and has 40 ratings, averaging 1.9 stars out of 5. What percent of the readers rated it, and what does the average say about the readers?
SolutionThe raters chose to click, a voluntary response sample of the readers, and they are
of them, fewer than one in a hundred.
Readers click when something moves them, annoyance or real pleasure; readers who found the article fine and moved on are under-counted. If annoyed readers are keener to rate than content ones, the average is pushed down, but the stars alone cannot say whether that is so. The method shows the sample is unreliable without showing which way it leans, so the 1.9 stars says little about most readers.
The first lunch period
A school has three lunch periods of 150 students each. The kitchen cooks the hot meal once each morning and serves it until it runs out. Desmond-Ray asks all 150 students in the first period: “This week, did the hot meal ever run out before you reached the front of the queue?” 12 say yes. Find the percent and the fraction of the school asked, and say which way the result is pushed.
Solution\displaystyle \frac{12}{150} = 0.08 = 8\% said yes. The school has 3 \times 150 = 450 students, and \displaystyle \frac{150}{450} = \frac{1}{3} of them were asked.
Desmond-Ray asked whoever was in front of him at his own lunch, so this is a convenience sample of the school, even though it is every first-period student. The two later periods had no chance to be in it, and they eat after the meal has been served for a while, so they are the students most likely to find it gone. The result is pushed down: 8\% is likely below the figure for the whole school. A third of the school is a lot of students, but it is the wrong third for this question.
Apples from the top of the crate
Fergal checks each crate of apples for bruising before paying. A crate holds 120 apples in 6 layers of 20. He checks the top layer and finds 1 bruised apple. Find the percent bruised in his sample. Then, to be more careful, he checks the top layer of each of 10 crates, 200 apples, finding 10 bruised. Does the larger sample fix the problem?
SolutionSamples need not be of people. The population is the apples in the crate, the top layer is the easiest part to reach, and so this is a convenience sample; \displaystyle \frac{1}{20} = 5\% of it is bruised.
The bottom five layers are missed. Lower apples carry the weight of the layers above, and a packer may put the best fruit on top, so the result is pushed down: the crate is likely worse than 5\%.
The larger check gives \displaystyle \frac{10}{200} = 5\% again. Every one of its apples is still a top-layer apple, so it answers “how bruised are top layers?” more firmly, which is not Fergal’s question. Only a sample that could reach every layer would fix it.
The cycling magazine’s reader poll
A cycling magazine prints a form: “Should the city spend more of its road budget on cycle lanes?” Readers post it back. 2{,}400 forms arrive, and 2{,}040 say yes. The headline reads “The city wants cycle lanes”. Find the percent saying yes, and judge the headline.
Solution\displaystyle \frac{2040}{2400} = 0.85 = 85\% of the forms say yes.
The headline is about the city, so the population is the city’s adults, and the forms pass two filters. First, only the magazine’s readers could answer: a convenience sample of the city, since the magazine asked the people it already reaches. A cycling magazine’s readers likely include far more cyclists than the city does, and cyclists have more reason than others to want cycle lanes. Second, only readers who cared enough to post the form answered: a voluntary response sample of the readers, in which the keenest cyclists are the likeliest to bother.
Both filters push toward yes, so 85\% is likely well above the city’s figure, by an amount the poll cannot tell. The 2{,}400 replies do not rescue it: more forms from the same keen readers would settle even more firmly near the readers’ answer. The headline is not supported.
Boundaries
Convenient does not always mean wrong. If the reached group matches the rest in the thing measured, there is no lean: shoppers at one entrance on a weekday morning are probably no more likely than others to be left-handed. But that belief rests on an argument from outside the data; the sample cannot show it.
The direction is not always knowable. The direction rule needs to know how the reached group differs from the missed one. When strong feelings could run either way, as with the article’s stars, all you can say is that strong feelings are over-counted.
The population has to be named first. Desmond-Ray’s survey is a convenience sample of the school but a complete count of the first lunch period.
This lesson is about who is in the sample, not what they say. The right people can still give untruthful answers or meet a question worded to push them; that is response bias and question wording. Groups with no chance of being chosen, and chosen people who never reply, are treated in undercoverage and nonresponse.
Where this goes next
The way out is to let chance, rather than ease or eagerness, decide who is in the sample: a simple random sample. A sample’s percent is a statistic estimating the population’s percent, the parameter. A biased method lands the statistic on the same side of the parameter again and again; a random one lets it fall on either side, and how widely it scatters is sampling variability.
Exercises
Naming the method
- Name each method as a convenience sample or a voluntary response sample. (a) A television cooking show invites viewers to text in their favorite pudding. (b) A teacher wants to know how long the students at her school spend on homework, and asks her own math class. (c) A town council’s website invites anyone who wishes to answer a survey about a new dog park. (d) A shoe shop asks the first 20 customers through the door on a Saturday how far they traveled.
- A made-up town council wants to know what percent of its adults exercise for at least three hours a week, and asks 40 adults leaving the town’s sports center one evening. 33 say they do. Find the percent, name the method, and say which way the result is likely pushed.
- A ranger wants the mean distance that visitors to a large park walk in a day, and asks the visitors in the parking lot beside the visitor center in the early afternoon. Name the method, a group likely missed, and the way the result is likely pushed.
Which way is the result pushed?
- A made-up cinema has stopped its late Monday showing, and asks customers to drop a card in a box saying whether it should come back. 80 cards are posted, and 64 say yes. Find the percent of cards saying yes, and explain why the customers as a whole may be less keen.
- Aurelia asks the 60 swimmers in a pool at six o’clock one weekday morning whether they find it too crowded, and 45 say it is never too crowded. Find the percent, and say which way the result is likely pushed and why.
- A made-up recipe page shows an average of 4.8 stars from 25 ratings, and has been visited 10{,}000 times. What percent of the visits ended in a rating? Explain why 4.8 stars may not describe the visitors.
- A shopper judges a punnet of strawberries by the ones on top, seen through the clear lid. Name the method, the part missed, and the way the judgment is likely pushed.
- Celestine wants to know what fraction of the students at her school walk to school, and asks the students waiting at the school bus stop one afternoon. Which way does her answer lean, and why would asking at the bus stop every afternoon for a week not fix it?
Counting the lean
- A made-up streaming service has 1{,}000 subscribers, and 200 of them dislike its new menu layout. Half of those who dislike it press the “Send feedback” button, and 1 in 20 of the other subscribers do. (a) How many replies come in? (b) What percent of the replies come from subscribers who dislike the layout, to the nearest whole percent? (c) What percent of all the subscribers dislike it? (d) With 2{,}000 subscribers, 400 of whom dislike the layout, and the same habits, what percent of the replies would come from those who dislike it?
- Barnaby asks 50 shoppers at the north entrance on Tuesday morning whether they would use late opening, and 9 say yes. On Wednesday and Thursday mornings he asks 200 more there, and 38 of them say yes. Find the percent saying yes in the first 50, the next 200, and all 250. Barnaby says the bigger sample makes his figure reliable. What did it do, and not do?
- At the school in Example 4, 30 of the 150 first-period students bring lunch from home, and 120 of the 300 students in the other two periods do. Find the percent for the first period, for the other two, and for the whole school. Which way would a first-period survey push the school’s figure?
- A made-up hotel leaves a comment card in every room. In one month it has 1{,}200 guests, and 360 of them had a complaint. 90 of the guests with a complaint fill in a card, and 42 of the other 840 do. (a) What percent of the cards come from guests with a complaint, to the nearest whole percent? (b) What percent of all the guests had a complaint? (c) How many times as likely to fill in a card was a guest with a complaint?
Reasoning
- Evadne says a convenience sample always gives the wrong answer. Describe a question for which a convenience sample would likely give a fair answer, and explain why you still could not be sure from the sample alone.
- Aurelia says every voluntary response sample is a kind of convenience sample. Explain what the two have in common, and what is different about where a voluntary response sample’s lean comes from.
- The station in Example 2 replaces phone calls with a form on its website, so that answering is easier. Explain why this is still a voluntary response sample, and whether it removes the lean.
Answers
- (a) Voluntary response: viewers choose whether to text. (b) Convenience: her class is at hand. (c) Voluntary response: anyone who wishes answers. (d) Convenience: the first through the door.
- \displaystyle \frac{33}{40} = 0.825 = 82.5\%. A convenience sample: adults leaving a sports center have just been exercising, so regular exercisers are over-counted and adults who rarely exercise are almost never there. The result is likely pushed up.
- Convenience sample. Visitors on long walks are likely still out on the paths in the early afternoon, so they are missed, while those near the visitor center then are more likely on short walks. The result is pushed down.
- \displaystyle \frac{64}{80} = 0.8 = 80\%. A voluntary response sample: customers who miss the late Monday showing have a reason to post a card, and customers who never went to it mostly do not. So the keen side is over-counted and the result is pushed up.
- \displaystyle \frac{45}{60} = 0.75 = 75\%. A convenience sample: swimmers who come at busy times, after school, after work and at weekends, are missed, and they are the ones who meet a crowded pool. The result is pushed toward “never too crowded”.
- \displaystyle \frac{25}{10000} = 0.0025 = 0.25\%. The raters chose themselves, a voluntary response sample of a quarter of one percent of the visits. They are the visitors moved to act, perhaps those who cooked the recipe and loved it, so 4.8 stars describes them, not the visitors as a whole.
- Convenience sample. The strawberries underneath are missed; they may be squashed, and the best may have been put on top, so the punnet likely looks better than it is.
- It leans toward “few walk”: students at the bus stop are mostly bus riders, and walkers are almost entirely missed. A week of asking there finds more bus riders, not more walkers, so it only settles more firmly on the bus stop’s answer.
- (a) 100 of the 200 reply, and \displaystyle \frac{800}{20} = 40 of the other 800 do: 140 replies. (b) \displaystyle \frac{100}{140} = \frac{5}{7}, about 71\%. (c) \displaystyle \frac{200}{1000} = 20\%. (d) 200 of the 400 who dislike it reply, and 80 of the other 1{,}600: \displaystyle \frac{200}{280} = \frac{5}{7}, about 71\% again. More subscribers, the same lean.
- \displaystyle \frac{9}{50} = 18\%, \displaystyle \frac{38}{200} = 19\%, and \displaystyle \frac{47}{250} = 18.8\% together. The bigger sample made the figure steadier for weekday-morning shoppers at the north entrance, but it still missed the shoppers who work weekday daytime hours, so the lean remains.
- \displaystyle \frac{30}{150} = 20\%; \displaystyle \frac{120}{300} = 40\%; \displaystyle \frac{150}{450} = \frac{1}{3}, about 33\%, for the whole school, between its parts. A first-period survey would push it down, to 20\%.
- (a) 90 + 42 = 132 cards, and \displaystyle \frac{90}{132} = \frac{15}{22}, about 68\%. (b) \displaystyle \frac{360}{1200} = 30\%. (c) A guest with a complaint filled one in with chance \displaystyle \frac{90}{360} = \frac{1}{4}, any other guest with chance \displaystyle \frac{42}{840} = \frac{1}{20}, and \displaystyle \frac{1}{4} \div \frac{1}{20} = 5: five times as likely, which lifts 30\% to 68\%.
- For example, whether shoppers at one entrance have a phone number ending in an even digit: there is no reason the door or the hour should be linked to the last digit of a phone number. But the missed shoppers were never asked, so the sample cannot show they match the reached ones; the trust rests on an argument from outside the data.
- In both, people are not chosen by chance, so whole groups can be over-counted or missed. A convenience sample leans because of where and when the collector looked. A voluntary response sample leans because of the respondents’ own choice, which is driven by how they feel about the question, so its lean is tied to the very thing being measured.
- Listeners still decide for themselves whether to answer, so it is still a voluntary response sample. Easier answering may bring in a few more listeners who do not mind, but objectors still have more reason to answer, so the lean remains.