Observational studies and experiments
Plants given fertilizer grew bigger. Did the fertilizer do that, or were those plants different to begin with?
Where you are 6 in Sampling and study design
Why this exists
Oktavia grows lettuce. She gives fertilizer to some of her plants, and those plants end up heavier. It is tempting to say the fertilizer did it. But suppose she gave the fertilizer to the plants that already looked strongest. Then the fertilized plants were ahead before any fertilizer touched them, and their extra mass proves nothing.
Every question of the form “does this cause that?” runs into the same wall. Two groups differ in the thing you care about, and they may also differ in a dozen things nobody measured. How the data was collected decides whether you can get past that wall. This lesson sorts studies into three kinds and says what each kind is allowed to conclude. All the studies and numbers in it are made up.
Definition
A sample survey asks a sample questions in order to describe a population: how many lettuces shoppers buy in a month, how many books customers read in a year. Its aim is a number for the whole population, which is the business of Populations, samples and census and simple random samples.
An observational study records what its subjects already do or already are, and imposes nothing. The researcher watches, measures or looks up records, and usually compares groups that formed on their own: people who keep a vegetable garden against people who do not.
An experiment imposes something on its subjects on purpose and then measures what happens. The thing imposed is a treatment, and whatever it is given to (people, plants, anything) are the subjects of the experiment.
One question separates the three: did the researcher decide who gets what? If yes, it is an experiment. If not, it is observational, and a survey is the kind of observational study whose aim is to describe a population from answers to questions.
Inside a study, the explanatory variable is the one that might explain or influence the other, and the response variable is the outcome measured. These are the same terms used for plotting paired data. In an experiment the explanatory variable is the treatment: fertilizer or none. In an observational study it is something the subjects brought with them: garden or no garden.
Three more words belong to experiments.
- A control group gets no treatment, or the usual one, so there is something to compare with. In medical trials it often gets a placebo, a dummy treatment that looks like the real one, so that simply believing you are treated affects both groups alike.
- Random assignment uses chance (shuffled cards, names from a hat, a random number generator) to decide which subject goes to which group.
- A randomized experiment is an experiment that uses random assignment.
Intuition
Figure 1 is Oktavia’s experiment done properly. She has 40 lettuce plants of one variety. She writes the numbers 1 to 40 on cards, shuffles them, and deals 20 to a fertilizer pile and 20 to a no-fertilizer pile. Chance, not Oktavia, decides which plant is in which group. Six weeks later she weighs every plant and compares the groups.
Some plants sit nearer the window, some grew from bigger seeds, some have better soil in their pots. Shuffling does not remove those differences; it shares them out. A sunny plant is as likely to land in one pile as the other, so each pile gets about half of the sunny plants, about half of the strong seeds, about half of everything, including things nobody thought to measure. The one difference not shared out by chance is the one Oktavia imposed: fertilizer.
Figure 2 is an observational study. Ottavio asks 200 adults whether they keep a vegetable garden and has each keep a food diary for a week. Seventy keep a garden and 130 do not. Nobody assigned the gardens: the people sorted themselves, which the dashed lines show.
Suppose the gardeners eat more vegetables. People who choose to garden may be people who already like vegetables, or who have more time at home, or more space. Each of those travels with the group, and none was shared out by chance.
Results
A well-run experiment with random assignment can show cause and effect. Well-run means the groups are treated alike apart from the treatment. If the groups then differ in the response by more than chance alone would produce, the treatment caused the difference.
An observational study can show an association only. The groups may differ in other ways, so a difference in the response cannot be put down to the explanatory variable. A variable that travels with both the explanatory and the response variable is a confounding variable, the subject of Correlation is not causation: lurking and confounding variables.
Random selection and random assignment answer different questions.
| answers the question | so the result | |
|---|---|---|
| random selection from a population | who does the result apply to? | can be extended to the whole population |
| random assignment to treatments | does the result show cause? | can be read as cause and effect |
A study can have both, either or neither; Scope of inference: random selection and random assignment sets out all four cases. Figure 3 shows a design with both.
When an experiment cannot be done, because imposing the treatment would be wrong or is impossible, an observational study is the only route, and its conclusion stays an association.
Why it is true
Before any treatment, the 40 plants differ in many ways. Take one of them, sunlight. Random assignment gives every plant the same chance of each group, whatever its sunlight, so the sunny plants split between the groups about evenly. The same holds, at the same time, for every other difference, measured or not. No difference can line up with the treatment except by chance, because the shuffle never looked at the plants.
After the treatment, the groups differ in two ways only: the treatment, and the leftover unevenness of the shuffle. That unevenness is chance, and how large chance differences tend to be can be worked out, by reshuffling, in Comparing two treatments by re-randomizing. A difference too big for chance is then down to the treatment. (A difference like that is called statistically significant, a term met again in later work.)
In an observational study the subjects formed the groups themselves, so whatever made them choose may be tied to the response: liking vegetables, time, space. A difference could come from the explanatory variable or from any of those, and the data cannot say which. Measuring more variables helps only with the ones measured. That is why cause needs assignment, not a bigger or more careful observation.
Worked examples
Same question, three designs
Each study is about lettuce. Is it a sample survey, an observational study or an experiment?
(a) A market asks a random sample of 200 shoppers how many lettuces they buy in a month, to estimate the mean for all its shoppers.
(b) A researcher visits 60 gardens, records whether each gardener uses fertilizer, and weighs a lettuce from each garden.
(c) Oktavia’s 40 plants are dealt by shuffled cards into fertilizer and no-fertilizer groups and weighed after six weeks.
SolutionAsk of each: did the researcher decide who gets what?
(a) Nobody is given anything, and the aim is one number describing the population of shoppers. It is a sample survey.
(b) The gardeners chose for themselves whether to use fertilizer; the researcher only recorded it. Nothing is imposed, so it is an observational study, comparing groups that formed on their own.
(c) Oktavia decided, by chance, which plants got fertilizer. A treatment was imposed, so it is an experiment, and a randomized one.
Studies (b) and (c) ask the same question, whether fertilizer makes lettuce heavier. Only (c) can answer it as cause and effect.
Reading an experiment
In Oktavia’s experiment the 20 fertilized plants had a mean mass of 310 grams after six weeks and the other 20 a mean of 262 grams. Name the explanatory and response variables, the treatments and the control group, find the difference in means, and say what she can conclude.
SolutionThe explanatory variable is whether a plant got fertilizer, the thing Oktavia changed. The response variable is the mass after six weeks, the thing she measured. The two treatments are fertilizer and no fertilizer, and the no-fertilizer plants are the control group, the baseline she compares with.
The difference in means is
Because the plants were randomly assigned, the groups were alike in everything except fertilizer, apart from chance. If 48 grams is more than chance alone would produce, Oktavia can conclude that the fertilizer caused the extra mass, for plants like hers grown the way hers were.
Reading an observational study
Ottavio’s 70 gardeners (Figure 2) ate a mean of 24 portions of vegetables in their diary week, and the 130 others a mean of 17. What can he conclude?
SolutionThe difference is 24 - 17 = 7 portions a week. The adults chose whether to keep a garden, so the study is observational. Ottavio can say that keeping a garden and eating vegetables are associated in his sample: the gardeners ate more, on average.
He cannot say gardening makes people eat more vegetables. The gardeners may differ from the others in ways that also affect eating: liking vegetables to begin with, cooking at home more, having more free time. Any of those could produce the gap with no help from the garden. The arrow could even point the other way, with a liking for vegetables leading people to start a garden.
Both kinds of randomness
A vet clinic has 3{,}000 clients with an appointment booked. Olimpia, who runs it, picks 100 of them at random from the booking list. She then randomly assigns 50 to get a text two days before their appointment and 50 to get no text, the clinic’s usual practice (Figure 3). In the text group 42 arrive on time; in the no-text group 31 do. Find each group’s percentage, and say what Olimpia can conclude and about whom.
Solutiona difference of 84 - 62 = 22 percentage points.
The two random steps do two jobs. Random assignment made the groups alike, apart from chance, in everything but the text, so, if 22 points is more than chance would give, the texts caused more on-time arrivals. Random selection made the 100 a fair picture of the 3{,}000, so the conclusion applies to all the clients with appointments booked, not only to these 100.
Had Olimpia asked for 100 volunteers instead, the assignment would still show cause, but only for people like the volunteers, who may be unusually organized. Had she let clients choose whether to sign up for texts, there would be no random assignment, and the gap would show an association only.
When an experiment would be wrong
A researcher wants to know whether breathing wood smoke at home harms children’s lungs. Why can she not run an experiment, and what can she do instead?
SolutionAn experiment would mean assigning some children to breathe smoke. Imposing a possible harm on people is wrong, so that experiment must not be run.
She can run an observational study: find homes that already burn wood and homes that do not, and measure the children’s lungs. If the children from wood-burning homes have weaker lungs, that is an association. Those homes may differ in other ways (older or damper houses, a different area), so the study cannot by itself show that the smoke is the cause.
A treatment with no fair comparison
Before the study in Example 4, Olimpia tried something simpler. Last February 64\% of the clinic’s clients arrived on time. In March the clinic texted every client, and 80\% arrived on time. A colleague says the texts raised on-time arrivals by 16 percentage points. What kind of study is this, and is the claim justified?
SolutionThe clinic imposed a treatment, the text, so by the definition above this is an experiment, but an uncontrolled one: a before-and-after comparison. It has no control group running at the same time and no random assignment: everyone in March got texts and nobody in February did.
The months differ by 80 - 64 = 16 percentage points, but they also differ in other ways: weather, school vacations, which clients happened to book. Any of these could change on-time arrivals, so the 16 points cannot be put down to the texts, and the claim is not justified. All the data show is that on-time arrivals were higher in the month with texts. Example 4 fixes exactly this: both groups run in the same weeks, and chance decides who gets a text.
Boundaries
Cause is about the treatment as given, to subjects like these. Oktavia’s result is about her fertilizer, at her dose, on her variety. On its own it says nothing about other doses or plants.
Random assignment removes systematic differences, not chance ones. With few subjects the groups can still differ by luck, which is why a difference has to be weighed against chance and why more subjects help. Experimental design takes this further, including blocking: grouping similar subjects before assigning.
Random assignment does not make a sample representative. Randomly assigned volunteers give a sound conclusion about cause for people like the volunteers. Extending a result to a population needs random selection, which convenience and voluntary samples do not have.
An association is still a real finding. “The gardeners ate more vegetables” is true of Ottavio’s sample. What his study cannot support is the word “because”.
Survey and observational study overlap. A survey imposes nothing, so it is observational; the label “survey” marks the aim, describing a population from answers to questions. Those answers can be bent by how the questions are asked, which is Response bias and question wording.
Imposing is the test, not the place. A study in a laboratory that only measures people is observational; a study in a garden that decides which plants get fertilizer is an experiment.
Where this goes next
Examples 2 and 4 leave open how big a difference has to be before chance is ruled out; Comparing two treatments by re-randomizing answers that by reshuffling the actual results many times. Correlation is not causation: lurking and confounding variables studies the hidden differences that stop an observational study showing cause, and Experimental design builds the tools of a good experiment: control, randomization, replication and blocking. The same caution carries into scatter plots and two-way tables: a pattern in data is an association, and whether it is also cause depends on how the data was collected.
Exercises
Naming the design
For Exercises 1 to 5, say whether the study is a sample survey, an observational study or an experiment, and name the explanatory and response variables where there are two.
- A chemist records, for each of 90 days, the pollen count and the number of hay fever tablets sold.
- A swimming coach randomly assigns 24 swimmers to one of two kicking drills for a month, then times each swimmer over 100 meters.
- A bookshop asks a random sample of 400 customers how many books they read last year, to estimate the mean for all its customers.
- Researchers test the hearing of 150 people who play in loud bands and 150 people who do not.
- Placido tosses a coin for each of his 60 basil plants to decide which get an extra hour of lamp light each day; 30 do. After a month he weighs the leaves of each plant. Also name the control group.
Groups and numbers
- In an experiment, 36 tomato plants are assigned at random, 18 to a new potting mix and 18 to the old one. The new-mix plants give a mean of 2.4 kilograms of tomatoes each and the old-mix plants 1.9 kilograms. Find the difference, and say what can be concluded if it is more than chance would produce.
- Olimpia repeats her study with 60 clients, randomly assigning 30 to a text and 30 to an email. Of the text group 27 arrive on time, and of the email group 21. Find each percentage and the difference.
- In a second garden study, Natalio finds that of 120 adults, the 45 who keep a garden eat a mean of 22 portions of vegetables a week and the others 18. Find the difference and say what it shows.
- Of Oktavia’s 40 plants, 10 stand on the shady side of the greenhouse. If the 40 are randomly split into two groups of 20, about how many shady plants would you expect in each group? Why only “about”?
Selection and assignment
For Exercises 10 to 12, say whether the study can show cause and effect, and who the result applies to.
- Ortensia picks 50 members at random from a running club’s list and records whether each stretches before runs and how many injuries each had last year.
- A clinic randomly assigns 60 volunteers to a new pillow or their usual one and records a neck-pain score after a month.
- A city picks 80 of its 6{,}000 bus-pass holders at random, randomly assigns 40 of them to an app showing live bus times and the other 40 to no app, and records how often each rides in a month.
Reasoning
- Someone argues: “The garden study in Exercise 8 had 120 people and Oktavia’s experiment only 40 plants, so the garden study is stronger evidence about cause.” Explain why the argument fails.
- Ottavio considers redoing his study with exactly 100 gardeners and 100 non-gardeners. Would equal groups let him conclude cause? Explain.
- Explain why no experiment can find out whether being born in winter affects adult height, and what kind of study can be used instead.
- A trial of a headache tablet gives its control group a sugar pill that looks identical, rather than nothing. Explain why.
- Suppose that, instead of shuffling, the fertilizer group were picked by hand so the two groups match in sunlight and seed source. Why is random assignment still better than careful matching?
Answers
- Observational study: nothing is imposed. Explanatory: pollen count; response: tablets sold.
- Experiment: the coach imposed the drills, by random assignment. Explanatory: which drill; response: the 100-meter time.
- Sample survey: questions to a random sample, to describe all customers. One variable, books read.
- Observational study: the researchers did not decide who plays in bands. Explanatory: playing in a loud band or not; response: hearing.
- Experiment. Explanatory: extra light or not; response: mass of leaves. The control group is the 30 plants with no extra light.
- 2.4 - 1.9 = 0.5 kilograms. The plants were randomly assigned, so if 0.5 kilograms is more than chance would produce, the new mix caused the extra yield, for plants like these grown this way.
- 27/30 = 90\% and 21/30 = 70\%, a difference of 20 percentage points. Here two treatments are compared directly: if the gap is more than chance, texts cause more on-time arrivals than emails, for clients like these 60.
- 22 - 18 = 4 portions a week. An association only: the adults chose whether to garden, so the groups may differ in other ways.
- About 5 in each, half of 10, since each shady plant is as likely to land in one group as the other. Chance can give 4 and 6, or 3 and 7, so not exactly 5.
- Association only, since stretching was each runner’s own choice; random selection means it applies to the whole club.
- Cause and effect, by random assignment, but only for people like these volunteers, since there was no random selection.
- Both: cause and effect, by random assignment, and it applies to all 6{,}000 pass holders, by random selection.
- Size does not decide cause. In the garden study people chose their group, so other differences travel with it however many are studied. Oktavia’s random assignment shares those differences out, which is what allows cause.
- No. Equal sizes change how many are in each group, not who they are. The gardeners still chose to garden and may still differ from the others in liking vegetables, time or space.
- Nobody can assign a birth season, so the treatment cannot be imposed. An observational study comparing adults born in different seasons can show an association only.
- Believing you have taken something can change how a headache feels. With a look-alike pill both groups believe they are treated, so that effect is the same in both, and a remaining difference is down to the tablet.
- Matching covers only what the picker thinks of and can see. Random assignment shares out unmeasured differences too, and stops the picker’s own choices, such as favoring healthy-looking plants, from lining up with the treatment.