Statistics · S.IC.B.5 · S.IC.B.3

Random assignment and confounding

An experimenter has to split the units into treatment groups somehow. Which ways of splitting can the comparison trust, and how do you check a split once it is made?

You need

Correlation is not causation: lurking and confounding variablesAlgebra 1 Control groups, placebos and blindingStatistics

Unlocks

Completely randomized design and replicationStatistics Scope of inference: random selection and random assignmentGrades 11 to 12

Where you are 7 in Experimental design

Statistics7

Sampling and study design6

Experimental design7

▴ 1 above Control groups, placebos and blinding Units, treatments, factors and levels

Random assignment and confounding

Completely randomized design and replication Scope of inference: random selection and random assignment ▾ Show all 7 (1 more)

Why this exists

Mariam runs a made-up poultry farm with 24 pens of laying birds, numbered 1 to 24 along the barn: pens 1 to 12 on the south side, 13 to 24 on the north side. She wants to know whether a new feed raises egg production compared with the standard feed, so 12 pens will get each. The pens are the experimental units, and the standard-feed pens are the control group, the baseline the new feed is compared against. All data in this lesson are made up.

Lasha, who hauls the feed, suggests the easy split: the truck parks at the south end, so give the new feed to pens 1 to 12. Tornike, who knows the birds, offers to pick the 12 pens that look healthiest. Both plans let something besides the feed decide who gets it, and that something can also affect laying. Observational studies and experiments explains why chance is the cure; this lesson is about carrying it out, checking it with numbers, and what it cannot do.

Definition

Random assignment decides which unit gets which treatment by a chance process that gives every possible division of the units into groups of the planned sizes the same chance. With 24 pens and two groups of 12, every set of 12 pens is equally likely to be the new-feed group. Three ways to do it:

Slips in a hat. Write each unit’s label on an identical slip, mix well, draw 12; those units get the new feed and the rest the standard feed.
A table of random digits. Label the units with the same number of digits (01 to 24), read the table in blocks of that many digits, and use a block only if it is a label not already chosen: skip repeats and blocks out of range. This is the reading taught in Simple random samples and the sampling frame.
Software. Shuffle the list of labels and take the first 12; or draw random integers from 1 to 24, skipping repeats, until 12 are chosen.

Confounding in an experiment happens when some variable other than the treatment differs between the treatment groups because of how the groups were formed, and that variable may also affect the response. Its effect and the treatment’s are then mixed together in the comparison and cannot be separated, the same trouble Correlation is not causation: lurking and confounding variables describes in observational data.

A baseline variable is something measured on each unit before treatment (here, eggs each pen laid in the week before the trial). The imbalance of an assignment on it is the difference between the two groups’ baseline means, new minus standard. After random assignment an imbalance is a chance imbalance.

Intuition

A random process has no preferences: the hat does not know which pens are on the sunny side. A plan that lets a person, a habit or a schedule decide can steer, even when nobody means it to, since a farm hand’s route follows the barn’s layout and “healthy-looking” also predicts laying.

Random assignment does not make the groups identical. It makes them differ only by chance, and chance can be measured. Figure 1 shows the assignment Mariam actually drew in Example 1: each feed got a mix of south-side and north-side pens, and the two baseline means sit within an egg of each other.

Figure 1One random assignment of the 24 pens (made-up data). Each group got a mix of high-laying south-side pens and lower-laying north-side pens, and the two means differ by less than one egg.

Results

No variable is tied to the treatment except by chance. Under random assignment each unit has the same chance of each group, whatever its traits, measured or not.

Chance imbalance centers on zero, and its typical size is known. Over all possible random assignments, the imbalance on a baseline variable averages 0, and its standard deviation is

\text{SD of the imbalance} = s\sqrt{\frac{1}{n_1} + \frac{1}{n_2}},

where n_1, n_2 are the group sizes and s is the standard deviation of the baseline over all n_1 + n_2 units, dividing by n_1 + n_2 - 1. Larger groups shrink it.

A plan that is not random carries no such guarantee. Its imbalance can be any size, and nothing about the plan says how large.

Why it is true

Each method is fair. Well-mixed identical slips give each remaining slip the same chance at every draw. In a random digit table each two-digit block 00 to 99 is equally likely, so among the blocks that are kept, each unused pen’s label is as likely as any other; skipping costs digits, not fairness. Either way every set of 12 pens is equally likely to be drawn.

The imbalance averages zero. Every unit has the same chance as any other of landing in the new group, so over all assignments the new group’s mean averages the mean of all the units. So does the other group’s mean, and their difference averages 0.

The spread formula. It is exact for any fixed set of units split at random, but its general proof needs the rules for the variance of random quantities, met later, so here it is checked, not proved: Example 5 lists every assignment of four pens, and the 2{,}704{,}156 assignments behind Figure 2 give exactly the same standard deviation as the formula.

Worked examples

Example 1

Drawing the pens with random digits

Mariam labels the pens 01 to 24 and reads this line of a random digit table, two digits at a time, to choose the 12 new-feed pens:

\begin{aligned} &07291\ 84453\ 16620\ 91307\ 25118\ 43922 \\ &06714\ 58810\ 23047\ 69152\ 11873\ 40296 \end{aligned}

Which pens get the new feed?

Solution

There are 24 pens, so every label needs two digits, and the line is read in pairs: 07, 29, 18, 44, 53, 16, 62, 09, 13, 07, 25, 11, 84, 39, 22, 06, 71, 45, 88, 10, 23, 04, 76, 91, 52, 11, 87, 34, 02. A pair is kept only if it is between 01 and 24 and not already chosen. Skipped as out of range: 29, 44, 53, 62, 25, 84, 39, 71, 45, 88, 76, 91, 52, 87, 34. Skipped as repeats: the second 07 and the second 11. Kept, in order:

07,\ 18,\ 16,\ 09,\ 13,\ 11,\ 22,\ 06,\ 10,\ 23,\ 04,\ 02.

That is 12 pens, so the reading stops. In order, pens 2, 4, 6, 7, 9, 10, 11, 13, 16, 18, 22 and 23 get the new feed; the other 12 get the standard feed.

Example 2

Checking the balance

Before the trial, Mariam recorded the eggs each pen laid in one week. Pens 1 to 12 (south), in order: 98, 103, 95, 101, 106, 97, 100, 104, 96, 99, 102, 107. Pens 13 to 24 (north): 88, 92, 85, 90, 94, 87, 91, 83, 89, 93, 86, 90. Find the imbalance for the assignment of Example 1, and for Lasha’s plan of pens 1 to 12.

Solution

The new-feed pens of Example 1 laid 103 + 101 + 97 + 100 + 96 + 99 + 102 = 698 eggs on the south side (pens 2, 4, 6, 7, 9, 10, 11) and 88 + 90 + 87 + 93 + 86 = 444 on the north side, so

\bar{x}_{\text{new}} = \frac{698 + 444}{12} = \frac{1142}{12} \approx 95.17 \text{ eggs}.

All 24 pens laid 1208 + 1068 = 2276 eggs (south row plus north row), so the standard-feed pens laid 2276 - 1142 = 1134, a mean of 1134/12 = 94.5 eggs. The imbalance is 95.17 - 94.5 \approx 0.67 eggs. The new feed got 7 south-side pens and 5 north-side, close to an even split (Figure 1).

Lasha’s plan puts the whole south row on the new feed: 1208/12 \approx 100.67 against 1068/12 = 89, an imbalance of 11.67 eggs before any feed is given. Every new-feed pen would be a south-side pen, so the side of the barn is confounded with the feed. If those pens later lay more, nobody could say how much came from the feed and how much from the side.

Example 3

How much imbalance to expect

The 24 baselines have standard deviation s \approx 6.93 eggs. Find the typical chance imbalance for two groups of 12, and judge both imbalances of Example 2 against it.

Solution

By the formula in Results,

\begin{aligned} \text{SD} &= 6.93\sqrt{\tfrac{1}{12} + \tfrac{1}{12}} \\ &= \frac{6.93}{\sqrt{6}} \approx 2.83 \text{ eggs}. \end{aligned}

Counting every assignment confirms the picture (Figure 2): of the \binom{24}{12} = 2{,}704{,}156 ways to choose the new-feed pens, about 0.66 give an imbalance within 2.83 eggs either way and about 0.95 within 5.66.

Figure 2Every possible assignment of the 24 pens, counted (made-up data). Chance imbalances of a few eggs are ordinary; the pens 1 to 12 plan sits at the far edge, where only 2 of the 2,704,156 assignments reach.

Mariam’s 0.67 eggs is about a quarter of a standard deviation: an ordinary draw. Lasha’s 11.67 is about 4.1 standard deviations. The south pens are the 12 highest baselines (the lowest south value, 95, beats the highest north value, 94), so 11.67 is the largest imbalance any assignment can have, and only 2 of the 2{,}704{,}156 reach it in size: Lasha’s plan and its mirror image. Chance would essentially never pick it; the plan picks it every time.

Example 4

Plans that are not random

For each plan, say whether it is random assignment and what could be confounded with the feed.

(a) Alternate: odd-numbered pens get the new feed.

(b) Tornike picks the 12 healthiest-looking pens for the new feed.

(d) The first 12 pens cleaned each morning get the new feed.

Solution

(a) Not random: once pen 1 is set, every pen is decided. On these baselines it happens to balance well: the odd pens laid 1130 eggs, a mean of 1130/12 \approx 94.17, and the even pens 1146/12 = 95.5, an imbalance of -1.33. But balance on one variable does not make a plan random. Anything in the barn with the same every-other pattern (windows above every second pen) lines up with the feed perfectly.

(b) Not random. Whatever Tornike reads as “healthy-looking” (alert birds, clean feathers) likely predicts laying too, so health is confounded with the feed and the new feed starts ahead.

(d) Not random. Cleaning order follows the layout of the barn, so this is close to Lasha’s plan, with the side of the barn riding along.

Example 5

Too few units

Tornike’s own small farm has four pens, with baselines 92, 96, 101 and 107 eggs, two to each feed. List every random assignment and its imbalance, and check the spread formula.

Solution

Choosing the 2 new-feed pens from 4 can be done \binom{4}{2} = 6 ways:

new feed	standard	imbalance
92, 96	101, 107	94 - 104 = -10
101, 107	92, 96	104 - 94 = 10
92, 101	96, 107	96.5 - 101.5 = -5
96, 107	92, 101	101.5 - 96.5 = 5
92, 107	96, 101	99.5 - 98.5 = 1
96, 101	92, 107	98.5 - 99.5 = -1

The imbalances average 0, as Results says. Their mean square is (2 \cdot 100 + 2 \cdot 25 + 2 \cdot 1)/6 = 252/6 = 42. The four baselines have mean 99 and squared deviations 49 + 9 + 4 + 64 = 126, so s^2 = 126/3 = 42, and the formula gives s^2\left(\tfrac{1}{2} + \tfrac{1}{2}\right) = 42, a match. The typical imbalance is \displaystyle \sqrt{42} \approx 6.48 eggs, and 4 of the 6 assignments miss by 5 eggs or more. A fair hat cannot balance two pens a group; that needs the replication of Completely randomized design and replication.

Example 6

Pens that drop out

In week two of Mariam’s trial an illness breaks out in the north corner, and pens 13, 18 and 23, all on the new feed, are taken out. Find the imbalance between the pens that remain.

Solution

Their baselines are 88, 87 and 86, totaling 261. The 9 new-feed pens left laid 1142 - 261 = 881 eggs, a mean of 881/9 \approx 97.89; the standard-feed group is unchanged at 94.5. The imbalance grows from 0.67 to 97.89 - 94.5 \approx 3.39 eggs, now more than one standard deviation from Example 3, and the remaining new-feed group leans to the south side (7 of its 9 pens).

The groups compared are no longer the ones chance made, and had the feed caused the illness, the dropouts would carry part of its effect. A sound report states every dropout and checks the balance of the units that remain.

Boundaries

Chance imbalance is never ruled out. Random assignment makes a large imbalance unlikely, not impossible. For a variable known to matter, such as the side of the barn, blocking splits it evenly on purpose and randomizes within each part.

Small groups stay unbalanced. The spread \displaystyle s\sqrt{1/n_1 + 1/n_2} shrinks only as the groups grow. With a handful of units per group, large imbalances are common (Example 5).

Only the assignment is protected. Random assignment balances what units bring with them at the start. It does nothing about what happens afterward: dropouts, units that do not take their treatment, or a measurer who knows the groups (which blinding guards against).

The process must be followed as drawn. Redrawing until the groups “look right”, or swapping a unit someone dislikes, hands the choice back to a person.

It answers “did the treatment cause it?”, not “for whom?”. Who the results apply to depends on how the units were chosen, the subject of Scope of inference: random selection and random assignment.

Where this goes next

Completely randomized design and replication builds a whole experiment on this step, with replication to shrink chance imbalance, and Blocking: randomized block and matched pairs designs balances a known variable by design. The same shuffle, applied to the response instead of a baseline, judges whether an effect is bigger than chance: Comparing two treatments by re-randomizing, and later the two-sample t-test.

Exercises

Carrying out the assignment

A second barn on Mariam’s farm has 16 coops, labeled 01 to 16, and 8 are to get a new lighting schedule. Read this line in pairs to choose them: 72110\ 34811\ 14069\ 30900\ 02158\ 71342.
Lasha keeps 30 laying birds in single cages labeled 01 to 30 and wants 5 for a calcium supplement. Use the line 41280\ 56328\ 19003\ 07712.
A calculator’s random integers from 1 to 24 come out 14, 3, 21, 14, 8, 24, 3, 17. Which 6 pens does it choose?
Describe how to assign the 24 pens with slips in a hat, and say what the slips and the mixing must be like for the method to be random.
Mariam labels the pens 1 to 24 and reads a digit table one digit at a time. Explain what goes wrong.

Is it random?

For Exercises 6 to 9, say whether the plan is random assignment and name anything that could be confounded with the treatment.

Lasha tosses a coin for each pen: heads, new feed; tails, standard.
Pens whose numbers end in 1, 2, 3 or 4 get the new feed.
The farm’s vet picks the 12 pens she thinks would benefit most.
New flocks arrive from two hatcheries, and the first 12 pens filled get the new feed.

Checking the balance

The 16 coops of Exercise 1 laid these eggs in the week before the trial, coops 01 to 16 in order: 42, 55, 38, 47, 51, 44, 60, 36, 49, 53, 40, 46, 58, 43, 50, 45. For the coops chosen in Exercise 1, find each group’s baseline mean and the imbalance, new minus standard.
The 16 baselines have s \approx 6.94 eggs. Find the typical chance imbalance for two groups of 8, and judge the imbalance of Exercise 10.
Four coops have baselines 40, 46, 52 and 62 eggs, two to each schedule. List all 6 assignments with their imbalances, and check that the mean square of the imbalances equals s^2\left(\tfrac{1}{2} + \tfrac{1}{2}\right).
Had Mariam used 48 pens, 24 per feed, with the same s \approx 6.93, what would the typical chance imbalance be? Compare with Example 3.
In the trial of Exercise 10, coops 03 and 11, both on the new schedule, are taken out. Find the new imbalance.
Suppose Mariam had used Lasha’s plan, and after a month the new-feed pens averaged 11.5 more eggs a week than the standard-feed pens. What can she conclude about the feed?

Reasoning

Tornike says he will choose the 12 pens “at random” by walking the barn and pointing. Explain why this is not random assignment.
After a random assignment, the new-feed birds turn out older on average than the standard-feed birds. Is the experiment flawed? What could have been done if age was known to matter?
Use the spread formula to explain why replication helps random assignment do its job.
Mariam’s assignment had an imbalance of only 0.67 eggs on the baseline. Does that show the groups are balanced on bird age? Explain.

Answers

Coops 11, 03, 14, 06, 09, 02, 15, 13. Pairs: 72 (out), 11, 03, 48 (out), 11 (repeat), 14, 06, 93 (out), 09, 00 (out), 02, 15, 87 (out), 13; that makes 8.
Cages 28, 05, 19, 30, 12. Pairs: 41 (out), 28, 05, 63 (out), 28 (repeat), 19, 00 (out), 30, 77 (out), 12.
Pens 14, 3, 21, 8, 24, 17: the second 14 and second 3 are skipped as repeats.
Write 1 to 24 on 24 identical slips, mix thoroughly, draw 12 without putting any back; those pens get the new feed. The slips must be the same size and feel, and mixed between draws, so that no slip is likelier than another to be drawn.
Single digits reach only pens 1 to 9 (and 0 is skipped), so pens 10 to 24 could never be chosen. Two-digit labels 01 to 24, read in pairs, give every pen the same chance.
Chance alone decides each pen, so nothing is confounded by the method. But the group sizes are left to chance (it could be 15 and 9), so it is not this lesson’s equal-split kind; drawing 12 slips is.
Not random. The sides split 6 and 6, but pens 1 to 4, 11 to 14 and 21 to 24 are exactly the pens at the ends of the two rows, so anything that varies from the middle to the ends of a row (drafts from the doors, light) is tied to the feed.
Not random. Her judgment of “would benefit” (younger, healthier, better laying) may predict laying, so it is confounded with the feed.
Not random. The first flocks may come from one hatchery, so hatchery is confounded with the feed.
New: (55 + 38 + 44 + 49 + 40 + 58 + 43 + 50)/8 = 377/8 = 47.125 eggs. Standard: (757 - 377)/8 = 380/8 = 47.5 eggs. Imbalance 47.125 - 47.5 = -0.375 eggs.
\displaystyle 6.94\sqrt{\tfrac{1}{8} + \tfrac{1}{8}} = 6.94/2 = 3.47 eggs. The imbalance -0.375 is about a tenth of that: an ordinary, small chance imbalance.
New 40, 46: 43 - 57 = -14; new 52, 62: 14; new 40, 52: 46 - 54 = -8; new 46, 62: 8; new 40, 62: 51 - 49 = 2; new 46, 52: -2. Mean square (2 \cdot 196 + 2 \cdot 64 + 2 \cdot 4)/6 = 528/6 = 88. Mean 50, squared deviations 100 + 16 + 4 + 144 = 264, s^2 = 264/3 = 88, and 88 \cdot 1 = 88.
\displaystyle 6.93\sqrt{\tfrac{1}{24} + \tfrac{1}{24}} = 6.93/\sqrt{12} \approx 2.00 eggs, down from 2.83: doubling both groups divides the typical imbalance by \displaystyle \sqrt{2}.
Their baselines are 38 and 40. The 6 remaining new-schedule coops average (377 - 78)/6 = 299/6 \approx 49.83 eggs, so the imbalance is 49.83 - 47.5 \approx 2.33 eggs.
Almost nothing. The new-feed pens laid 11.67 more eggs a week before any feed was given (Example 2), so a gap of 11.5 afterward is what the side of the barn alone would produce. The feed’s effect is confounded with the side and cannot be estimated.
Pointing is not a chance process: habits (pens near the aisle, lively birds) steer it, and nothing gives every set of 12 the same chance.
Not flawed: chance imbalances are expected on some variable, and the report should note it. Blocking by age (randomizing separately within older and younger flocks) would have balanced it.
The typical chance imbalance is \displaystyle s\sqrt{1/n_1 + 1/n_2}, which shrinks as the groups grow. With many units per group, chance differences between the groups are small, so the groups are alike and a difference in response can be credited to the treatment.
No. Each variable has its own chance imbalance, and a small one on egg counts says nothing directly about age. Random assignment makes a large age imbalance unlikely, and only measuring age would show how large it is.