Blocking: randomized block and matched pairs designs
Some units are known to differ before an experiment starts, like vans that crawl downtown and vans that cruise the highway. How can the treatments be compared so that this known difference stays out of the comparison?
Where you are 7 in Experimental design
Why this exists
At the end of Completely randomized design and replication, one shuffle sent 6 of the 12 guitar players to one group and 2 to another. Nothing was unfair about that, but a variable known to matter was left to chance, and its variation swelled the noise. When such a variable is known before the experiment starts, it can be handled on purpose.
Ilinca runs a made-up delivery fleet of 24 vans and wants to compare three tire treads by fuel economy, in miles per gallon (mpg): the standard tread now on the vans (the control), a low-resistance “eco” tread, and a deep “grip” tread. Before she starts, she knows that route matters far more than tread: downtown vans stop at every corner and get about 14 mpg, highway vans about 23. If a shuffle put most highway vans on one tread, that tread would look good for a reason that has nothing to do with tires. Even with even luck, a 9 mpg spread between routes can bury a tread effect of 1 mpg.
The fix is to group the vans by route first and randomize inside each group. Day 1 builds this randomized block design; Day 2 takes it to blocks of two, the matched pairs design. All data in this lesson are made up.
The pieces it rests on: units and treatments say what gets what, a Control groups, placebos and blinding gives the baseline, and Random assignment and confounding lets chance form the groups so that they differ only by chance.
Definition
A block is a group of experimental units that are alike in some variable expected to affect the response, formed before treatments are assigned. The blocking variable is not a treatment: the experimenter does not impose it and is not studying it, only keeping its effect out of the comparison.
In a randomized block design, the units are first divided into blocks, and then the treatments are assigned at random separately within each block, by its own chance process. Usually each treatment appears equally often in every block. Treatments are compared within blocks, and those comparisons are combined.
A matched pairs design is a randomized block design for two treatments with blocks of size two. It comes in two forms.
- Paired units. Units are matched in pairs that are alike (twins, or two vans on the same route of the same model year), and in each pair a coin toss decides which unit gets which treatment.
- Each unit gets both. Every unit receives both treatments, one after the other, and chance decides the order for each unit. The unit is its own match.
Either way the data become one number per pair, the difference
summarized by the mean difference \bar{d} and the standard deviation of the differences s_d.
Intuition
Figure 1 outlines Ilinca’s design. In the outline of Completely randomized design and replication, one split formed every group at once. Here the vans are first grouped by route (not at random: route is a fact about each van), and then four separate random splits run, one per route.
Inside one route the vans are alike, so a difference between treads within that route is not muddied by route. Every tread meets every route equally often, so no tread can be favored by its routes. Figure 2 shows what this buys. In the top panel, each tread’s eight vans spread over nine mpg because of their routes, and the treads overlap almost completely. In the bottom panel each van’s route mean has been subtracted; what is left is the tread effect plus a little noise, and the treads separate.
Sampling has the same move: a Stratified random sample groups the population into alike strata and draws at random inside each. Strata steady an estimate of a population value; blocks steady a comparison of treatments.
Results
Day 1: blocks
What blocking removes. A comparison of treatments made within a block contains no difference between blocks. Combined over blocks, it is free of the blocking variable, of its bias and of its noise.
What randomizing within blocks does. Every other variable is still left to chance inside each block, so within a block the treatment groups differ only by chance.
Block what you can, randomize what you cannot. A variable that is known before the experiment and likely to affect the response is a candidate for blocks. Everything else, known or not, is handled by random assignment.
Day 2: matched pairs
Work with the differences. The n differences d are one column of data. Their mean \bar{d} estimates the treatment effect, and their spread s_d is the noise it must beat. When the matching is good, s_d is far smaller than the spread of either treatment’s responses.
Why it is true
Think of each response as three parts added together:
Take two units in the same block, one on treatment A and one on B. Their difference is
and the block level has canceled, however large it was. Averaging such differences over blocks shrinks the chance parts toward zero, and the block levels were never there to begin with. In a completely randomized design two units being compared can come from different blocks, so their difference also carries (\text{level}_1 - \text{level}_2): noise at best, and bias if the luck is lopsided. The assignment inside each block must still be random. If a person chose, the better units in a block could be steered to one treatment, and the chance parts would no longer average out.
Worked examples
Day 1: writing out the block design
Ilinca’s 24 vans run four routes, 6 vans on each: downtown, suburban, rural and highway. Write out a randomized block design comparing the standard, eco and grip treads, and count the possible assignments.
SolutionUnits: the 24 vans. Treatments: standard (the control), eco and grip.
Blocks: the four routes, formed before anything is assigned, because route is expected to affect fuel economy strongly. Route is a fact about each van, not something Ilinca assigns, so it is a block, not a treatment.
Assignment: within each route, label the 6 vans and shuffle them; the first 2 get standard, the next 2 eco, the last 2 grip. Four routes, four separate shuffles, so each route has 2 vans per tread and each tread 8 vans in all.
Response: mpg over one month, from fuel and odometer records, with drivers, loads and the month the same for every tread.
Comparison: each tread against standard within each route, combined over routes.
Within one route, the number of ways to split 6 vans into three labeled groups of 2 is
and the four shuffles are separate, so there are 90^4 = 65{,}610{,}000 assignments. A completely randomized design of 24 vans into three groups of 8 allows \displaystyle \frac{24!}{8!\,8!\,8!} = 9{,}465{,}511{,}770. Blocking keeps only the assignments that give every route 2 vans per tread, which rules out every lopsided one.
Day 1: the block differences drop out
Ilinca’s results, in mpg:
- downtown: standard 13.6, 14.0; eco 14.9, 14.7; grip 13.5, 13.3
- suburban: standard 16.9, 16.7; eco 17.8, 17.9; grip 16.3, 16.4
- rural: standard 19.7, 20.0; eco 20.7, 20.8; grip 19.5, 19.3
- highway: standard 22.8, 22.9; eco 23.9, 23.7; grip 22.4, 22.3
Compare eco with standard, first ignoring routes, then within routes.
SolutionIgnoring routes. The eight standard vans average \bar{x}_{S} = 18.325 mpg and the eight eco vans \bar{x}_{E} = 19.3 mpg, so eco leads by 0.975 mpg. But the eight values on one tread have a standard deviation of about 3.6 mpg, nearly all of it route. With pooled s \approx 3.59, the chance gap between two groups of 8 (the rough yardstick of Completely randomized design and replication, not a test) is about
mpg, and 0.975 is only about 0.54 of it: on these numbers alone, an ordinary chance difference.
Within routes. The route means are exactly 14, 17, 20 and 23 mpg. Eco minus standard in each route is downtown 14.8 - 13.8 = 1.0, suburban 17.85 - 16.8 = 1.05, rural 20.75 - 19.85 = 0.9 and highway 23.8 - 22.85 = 0.95.
These four average 0.975, the same estimate, because every tread met every route equally often. But they range only from 0.9 to 1.05. Subtracting each van’s route mean (Figure 2, bottom) leaves values with a pooled standard deviation of about 0.113 mpg, so the same rough chance gap drops to about 0.113 \times \tfrac{1}{2} \approx 0.057 mpg, and 0.975 is about 17 such gaps. The same 0.975 mpg that was lost in the route-to-route noise now stands far clear of it.
Day 1: telling the designs apart
Say whether each plan is completely randomized, a randomized block design, or neither.
(a) Doina splits the 24 vans into the 12 with automatic and the 12 with manual transmissions, and within each set draws names for 4 standard, 4 eco and 4 grip.
(b) Sorin assigns all 24 vans by one shuffle, 8 per tread, and after the month sorts the results by route.
(c) Every highway van gets eco, every downtown van standard, and the rural and suburban vans are shuffled among the three treads.
SolutionAsk of any description: is there a grouping before assignment, and a separate chance process inside each group?
(a) Randomized block design, blocked on transmission: the groups are formed first, with a separate random draw in each.
(b) Completely randomized. Sorting results by route afterward is a way of looking at data, not blocking; the assignment ignored route, so one tread may hold more highway vans than another.
(c) Neither. Highway and downtown vans get their treads by choice, and there tread and route are confounded: eco would look good because highway vans get better mileage anyway. A block must contain every treatment, assigned by chance.
Day 1: choosing what to block on
Ilinca considers four variables for her blocks: (a) route, (b) paint color, white or gray, (c) each van’s mpg last month, before the test, (d) tread. Which make good blocks?
SolutionA blocking variable must be known before assignment and should be strongly related to the response.
(a) Good: route drives fuel economy, as Example 2 showed.
(b) Useless. There is no reason paint color affects mileage, so a white block and a gray block would have about the same mean and subtracting it would remove nothing. Treatments are still randomized within each color, so it does no harm to fairness, but it buys nothing.
(c) Very good, and it captures route along with much else (driver habits, engine condition). Ilinca could sort the vans by last month’s mpg and form blocks of the 6 highest, the next 6, and so on. It is measured before the test, so it is not the response.
(d) Impossible. Tread is the treatment, and a block must contain all the treatments.
Block on what is known to matter; whatever is left, known or not, is handled by the randomization inside the blocks.
Day 2: matched vans, a coin toss each
Doina tests eco against standard on 12 pairs of vans, each pair matched on route and model year; in each pair a coin toss decides which van gets eco. One month’s mpg, pairs 1 to 12 in order:
- eco: 15.2, 18.4, 21.0, 23.9, 16.1, 19.3, 13.9, 22.4, 17.1, 20.6, 24.5, 14.8
- standard: 14.3, 17.5, 20.3, 22.6, 15.4, 18.1, 13.2, 21.6, 16.5, 19.4, 23.8, 13.7
Find the differences, \bar{d} and s_d, and compare them with the spread of each treatment’s values.
SolutionWith d = \text{eco} - \text{standard}, the twelve differences are 0.9, 0.9, 0.7, 1.3, 0.7, 1.2, 0.7, 0.8, 0.6, 1.2, 0.7, 1.1. They total 10.8, so \bar{d} = 10.8/12 = 0.9: on average the eco van got 0.9 mpg more than its matched standard van, with s_d \approx 0.24 mpg.
The eco values alone have standard deviation about 3.60 mpg and the standard values about 3.56, because the pairs run from downtown to highway. That spread is in each column but not in the differences: the two vans of a pair share a route, so it cancels pair by pair. Treated as two unrelated groups of 12, the chance gap would be about \displaystyle 3.58\sqrt{2/12} \approx 1.46 mpg, and 0.9 would sink in it. As one column of differences, the chance error of \bar{d} is about \displaystyle s_d/\sqrt{12} \approx 0.07 mpg, and 0.9 is far beyond it. Turning \bar{d} and s_d into an interval or a test is Paired data: interval and test for a mean difference.
Day 2: one van, both treads, random order
Sorin uses the other form. Each of his 12 vans runs one month on eco and one on standard. Six vans, drawn from a hat, run eco first; the other six run standard first. The second month was warmer, and warm weather alone raises mileage. Results as (eco, standard) in mpg:
- eco first: (16.3, 16.0), (19.6, 19.5), (14.6, 14.2), (21.8, 21.8), (17.8, 17.6), (23.3, 23.1)
- standard first: (17.0, 15.3), (22.3, 20.4), (19.8, 18.2), (24.5, 22.5), (15.6, 13.8), (20.8, 19.0)
Find \bar{d} (eco minus standard) for each order group and overall, and explain what randomizing the order did.
SolutionEco-first vans: d = 0.3, 0.1, 0.4, 0.0, 0.2, 0.2, total 1.2, mean 0.2. Standard-first vans: d = 1.7, 1.9, 1.6, 2.0, 1.8, 1.8, total 10.8, mean 1.8. Overall, \bar{d} = (1.2 + 10.8)/12 = 1.0 mpg.
The groups differ because the warm month helped whichever tread came second. For an eco-first van d is about (tread effect) minus (warm lift); for a standard-first van, (tread effect) plus (warm lift). With six of each, the lift cancels in the overall mean, which estimates the tread effect, 1.0 mpg, and half the gap between the groups, (1.8 - 0.2)/2 = 0.8 mpg, estimates the warm lift. Had every van run standard first, every d would sit near 1.8, and the warm month would be credited to the eco tread: order and treatment confounded. Random order keeps order out of the comparison, just as random assignment within a block keeps everything else out. The van-to-van spread, from 13.8 to 24.5 mpg, is gone entirely, since each van is compared only with itself.
Day 2: two treads on one van
Sorin now studies tread wear. On each of 12 vans he fits eco on one rear wheel and standard on the other, a coin toss deciding which goes on the left. After 20{,}000 miles he measures the tread depth each tire lost. What design is this, what are the blocks, and why the coin toss? Could the mileage study have been run this way?
SolutionIt is a matched pairs design. Each van is a block of two tire positions that share a driver, a route, a load and every mile; the comparison for a van is its difference d, standard wear minus eco wear, and the analysis uses the 12 differences.
Take it as a premise of this made-up study that the two sides of a van may wear differently (one side runs nearer the curb, where the road slopes), by an amount nobody knows. The coin toss keeps side from being confounded with tread: whatever side does, it is as likely to favor eco as standard. It might land 8 left and 4 right; that is chance imbalance, fair and allowed for.
Mileage could not be studied this way. Fuel economy belongs to the whole van, so one van with two treads gives one mpg that cannot be split between them. That is why mileage needed matched vans (Example 5) or separate months (Example 6).
Boundaries
Blocks must matter. Blocking helps only if the blocking variable is related to the response. On an unrelated variable the blocks have about equal means and nothing is removed.
Blocks are formed before assignment, every block holds every treatment, and the assignment inside each block is random. Sorting the results by a variable afterward is not blocking.
A block is not a treatment. The blocking variable is not assigned by chance, so the experiment shows that treatments cause differences, not that routes do. Differences between blocks are observed, not tested by the design.
Each unit gets both, only if the unit returns to where it started. The first treatment must leave no trace when the second begins: a driving course cannot be untaken. The order must be randomized, or anything that changes with time (weather, practice, fatigue) is confounded with treatment.
Matching is only as good as the match. Pairs alike only in an irrelevant way give differences no steadier than two separate groups.
It controls only what was blocked. Everything else rests on randomization within blocks, and who the results apply to depends on how the units were chosen (Scope of inference: random selection and random assignment).
Where this goes next
Paired data: interval and test for a mean difference takes the column of differences from a matched pairs design and builds an interval and a test for the mean difference. The shuffling of Comparing two treatments by re-randomizing carries over to blocks: shuffle the treatment labels only within each block, which for a pair means flipping the sign of each difference at random. Designs with several treatments and many blocks are analyzed later with methods that split the variation into its block, treatment and chance parts.
Exercises
Day 1: randomized block designs
- A second depot has 18 vans on three routes, 6 per route, and will test the same three treads with 2 vans per tread on each route. (a) How many assignments does the block design allow? (b) How many would a completely randomized design with 6 vans per tread allow?
- Doina will compare two air filters on 20 vans, 10 diesel and 10 gasoline, by one month’s mpg. Write out a randomized block design.
- Sorin compares two route-planning apps on 12 vans, 4 per route, 2 per app on each route. Minutes per delivery stop: city, app A 6.1, 5.9, app B 5.4, 5.6; suburban, app A 4.3, 4.5, app B 3.9, 3.7; rural, app A 8.2, 8.0, app B 7.7, 7.5. Find app A minus app B within each route and the mean of the three, and compare their spread with the spread of the six app A values.
- From Example 2’s data, find grip minus standard within each route and the mean of the four.
For Exercises 5 to 7, say whether the plan is completely randomized, a randomized block design, or neither.
- 30 vans are sorted into 10 small, 10 medium and 10 large; within each size, names are drawn for 5 on each of two brake pads.
- All 30 vans are shuffled into two groups of 15; afterward the analyst counts how many large vans landed in each group.
- Drivers choose which of two seat cushions to use, and the results are reported separately for the day and night shifts.
- Doina tests two ways of loading packages into a van; the response is minutes to load. Which make good blocks: (a) the loading dock, as docks differ in layout; (b) the loader’s shoe size; (c) the loading method; (d) the van’s size?
Day 2: matched pairs
- Ilinca tests a roof wind deflector on 6 pairs of vans matched by route; a coin toss in each pair picks the van that gets it. Mpg with the deflector: 18.2, 21.5, 15.0, 24.1, 16.8, 19.9; without, in the same pair order: 17.4, 20.9, 14.1, 23.5, 16.3, 19.1. Find the six differences and \bar{d}.
- For Exercise 9, find s_d and the standard deviation of the six deflector values, and explain why they differ so much.
- In another each-van-gets-both run, the vans with eco first average \bar{d} = 0.4 mpg (eco minus standard) and the equally many with standard first average 1.4. Estimate the tread effect and the second month’s lift.
- In Exercise 11, if every van had run standard first, about what would \bar{d} have been, and why would it mislead?
- (a) How many assignments do the 12 coin tosses of Example 5 allow? (b) In how many ways can 6 of 12 vans be chosen to run eco first?
- Name the design: (a) ten pairs of drivers matched on years of experience, a coin toss in each pair deciding who takes an eco-driving course, response next month’s mpg; (b) twelve vans each run eco in March and standard in April.
- Design an each-unit-gets-both study for 16 drivers rating the comfort of two seat cushions, one week on each, on a scale of 1 to 10.
Reasoning
- Why must blocks be formed before treatments are assigned?
- Explain “block what you can, randomize what you cannot” for Ilinca’s fleet, naming one variable of each kind.
- Explain why a matched pairs design is a randomized block design, and name the block in each form.
- Give a treatment for which the each-unit-gets-both form cannot work, and say what to use instead.
- How is blocking in an experiment like stratifying a sample, and how do their purposes differ?
Answers
- (a) 90^3 = 729{,}000. (b) \displaystyle \frac{18!}{6!\,6!\,6!} = 17{,}153{,}136.
- Units: the 20 vans. Treatments: filter A and filter B. Blocks: engine type, formed first, since diesel and gasoline vans differ in mpg. Within the diesel block shuffle the 10 vans, first 5 get A, the rest B; a separate shuffle does the same in the gasoline block. Response: one month’s mpg. Compare A with B within each engine type, then combine.
- City 6.0 - 5.5 = 0.5, suburban 4.4 - 3.8 = 0.6, rural 8.1 - 7.6 = 0.5; mean 1.6/3 \approx 0.53 minutes: app B is faster by about half a minute per stop. The differences vary by only 0.1, while the app A values run from 4.3 to 8.2, a range of 3.9 minutes, almost all of it route.
- Downtown 13.4 - 13.8 = -0.4, suburban 16.35 - 16.8 = -0.45, rural 19.4 - 19.85 = -0.45, highway 22.35 - 22.85 = -0.5; mean -0.45: grip costs about 0.45 mpg.
- Randomized block design, blocked on van size.
- Completely randomized: counting afterward is not blocking.
- Neither: drivers chose, so whatever makes a driver choose a cushion is confounded with it; the shift grouping comes after the fact.
- (a) Good: layout affects loading time and is known beforehand. (b) No: nothing links shoe size to time, so nothing would be removed. (c) Impossible: it is the treatment. (d) Good: bigger vans take longer to load.
- 0.8, 0.6, 0.9, 0.6, 0.5, 0.8; total 4.2, so \bar{d} = 0.7 mpg.
- Deviations from 0.7 square to a total of 0.12, so \displaystyle s_d = \sqrt{0.12/5} \approx 0.15 mpg. The deflector values have mean 19.25 and s \approx 3.29 mpg. They differ by route; the two vans of a pair share a route, so the differences do not.
- Tread effect (0.4 + 1.4)/2 = 0.9 mpg; second-month lift (1.4 - 0.4)/2 = 0.5 mpg.
- About 0.9 + 0.5 = 1.4 mpg, crediting the second month’s lift to eco: order and tread confounded.
- (a) 2^{12} = 4{,}096. (b) \binom{12}{6} = 924.
- (a) Matched pairs, paired units. (b) Each unit gets both, but the order is fixed, so month is confounded with tread; it needs random order.
- Each driver uses both cushions, one week each. Draw 8 names from a hat to use cushion A first; the other 8 use B first. Record the rating after each week, find d = A - B for each driver, and work with the 16 differences. Random order keeps weather or growing used to the seat from favoring one cushion.
- Blocks have to shape the assignment so that every treatment gets the same mix of each block; groupings made afterward cannot undo a lopsided assignment.
- Route is known before the test and affects mpg, so block on it. A driver’s lead foot or a van’s hidden engine wear cannot be listed in advance, so randomizing within routes balances them by chance.
- Each pair is a block of two alike units, and the two treatments are assigned at random within it. Paired units: the block is the matched pair. Each unit gets both: the block is the unit itself.
- An eco-driving course cannot be untaken, so a driver cannot later act as the untrained version. Use matched pairs of drivers with a coin toss, or a block design.
- Both group alike members first and use chance inside each group. Strata steady an estimate of a population value by sampling within groups; blocks steady a comparison of treatments by assigning within groups.