Statistics · S.IC.B.3 · S.IC.B.5

Completely randomized design and replication

You have units, treatments and a way to assign by chance. How do you build a whole experiment from them, and how many units does each treatment need before a real difference shows through the noise?

You need

Random assignment and confoundingStatistics

Unlocks

Blocking: randomized block and matched pairs designsStatistics

Where you are 7 in Experimental design

Statistics7

Sampling and study design6

Experimental design7

▴ 2 above Units, treatments, factors and levels Random assignment and confounding

Completely randomized design and replication

Scope of inference: random selection and random assignment Blocking: randomized block and matched pairs designs

When you reach for it

You want to know whether some treatments cause a difference in a response, you can impose the treatments yourself, and you know of nothing about the units that should be handled first. The pieces are already in place: the units and treatments, a control group to compare against, and Random assignment and confounding, which uses chance so that the groups differ only by chance. A completely randomized design is the plainest way to put these together into one experiment, and replication is what lets its result be judged.

Khoa teaches beginners at a made-up music school and wants to know how adults best learn to switch between ukulele chords. All data in this lesson are made up.

The procedure

The design. In a completely randomized design, all the units are assigned to the treatments by chance alone, with no grouping of the units beforehand. Usually the group sizes are planned and one chance process makes every way of splitting the units into groups of those sizes equally likely; letting chance decide each unit on its own, as a die roll does, also qualifies, though the group sizes then come out unequal (Example 4). Each unit then receives its treatment, the response is measured, and the groups are compared. Written out, a design names five things, and each has a reason.

The units, so the reader knows what was compared. Each unit must be able to receive any treatment, and no unit’s treatment may spill onto another.
The treatments, including the control. Without a control there is nothing to measure a gain against; with it, every other group is judged by its difference from the control.
The random assignment of all units at once. One shuffle of the whole list forms every group, so no trait of a unit can steer it toward a treatment. How to carry out the shuffle, and why it balances the groups, is in Random assignment and confounding.
The response, said precisely: what is measured, when, and by whom. Everything else that could affect the response is kept the same for every unit, so it cannot differ between groups.
The comparison, usually of group means: each treatment’s mean against the control’s mean.

The design is drawn as an outline: the units on the left, a split marked “random assignment”, one box per group with its size and treatment, and the groups joining at the comparison (Figure 1).

Figure 1The outline of Khoa’s completely randomized design (a made-up study). One random process forms all three groups at once.

Replication has two senses, and the design needs both.

Within the experiment: several units get each treatment. Units differ from one another even under the same treatment, and only by watching several units on one treatment can that unit-to-unit variation be seen and measured. A difference between treatments counts as evidence only when it is large next to that variation. With one unit per treatment, there is no way to tell a treatment effect from one unit simply being better than another.
Of the experiment: the whole study is carried out again, with new units, often by other people in another place. One experiment can be unlucky in its random assignment, or depend on something peculiar to its setting; a result that holds up when repeated is far more trustworthy.

Why more units help. A group mean is pulled around by which units happen to land in the group. The more units it averages, the more their personal ups and downs cancel, so the mean settles closer to what the treatment really does. Example 3 measures this: in a chance model, the typical chance error of a group mean is cut in half each time the group size is multiplied by four. A real difference between treatments stays the same size while the chance noise around it shrinks, so the difference becomes easier to see.

Worked examples

Example 1

Writing out the design

Khoa has 30 adult beginners. He wants to compare two ways of practicing chord changes against the school’s usual way: a rhythm app that listens and flashes each change on time, and a metronome whose speed is stepped up each day. The usual way is practicing from a printed chord chart. Write out a completely randomized design.

Solution

Units: the 30 beginners. Each practices alone at home, so one learner’s method cannot affect another’s.

Treatments: rhythm app, stepped metronome, and the chord chart. The chart is the control, the usual treatment, since the question is whether a new method beats what learners do now.

Assignment: with 30 units and 3 treatments, 10 per treatment keeps the replication equal. Label the learners 01 to 30, shuffle the whole list once, and give the first 10 the app, the next 10 the metronome and the last 10 the chart. Every split into three groups of 10 is then equally likely; there are

\binom{30}{10}\binom{20}{10} = 5{,}550{,}996{,}791{,}340

of them. Assigning all 30 in one process, rather than letting, say, the morning class fill the app group first, is what makes the design completely randomized.

Response: after three weeks of 15 minutes’ practice a day, each learner is filmed switching between the C and G chords for one minute, and a judge who does not know the groups counts the clean changes. The response is measured once per learner. Practice time, the ukulele model and the chord pair are the same for everyone.

Comparison: the mean number of changes per minute in each group, each new method against the chart.

Example 2

Reading the results

Khoa’s results, in clean changes per minute:

app: 31, 26, 35, 29, 33, 24, 38, 30, 27, 32
metronome: 28, 34, 25, 30, 22, 33, 29, 27, 36, 31
chart: 24, 29, 21, 27, 30, 19, 26, 23, 28, 25

Find the three group means and the differences from the control. The standard deviations within the groups are about 4.25, 4.25 and 3.52; use a pooled value s \approx 4.02 to judge which differences are bigger than chance.

Solution

The totals are 305, 295 and 252, so the means are \bar{x}_{\text{app}} = 30.5, \bar{x}_{\text{met}} = 29.5 and \bar{x}_{\text{chart}} = 25.2 changes per minute. Against the control, the app group averaged 30.5 - 25.2 = 5.3 more changes and the metronome group 29.5 - 25.2 = 4.3 more. The two new methods differ by 1.0.

Replication is what lets these be judged. Within each group the learners spread over about 4 changes per minute, all on the same method, and that spread is the chance part. If the methods made no difference at all, each score would just be that learner’s own level, and two random groups of 10 would still have means differing by roughly

s\sqrt{\tfrac{1}{10} + \tfrac{1}{10}} \approx 4.02\sqrt{0.2} \approx 1.80

changes per minute. This is the spread formula of Random assignment and confounding, now applied to the response, with one change: there s was the spread of all the units together, dividing by n_1 + n_2 - 1, but here the spread of all 30 scores together would also hold any real gap between methods. So s is taken from the spread within the groups, where everyone had the same method: each group’s variance (dividing by n - 1 = 9), averaged over the three groups, then the square root. The result is a rough yardstick, not a test. The app’s 5.3 is about 2.9 of these chance gaps and the metronome’s 4.3 about 2.4, both larger than chance usually makes. The 1.0 between app and metronome is about 0.56 of a gap, an ordinary chance difference. So the data point to both new methods beating the chart, with no clear winner between them. Settling such questions properly is the work of Comparing two treatments by re-randomizing.

Example 3

Watching the noise shrink

Nhung builds a chance model to see what group size does. In it, the app really adds 4 changes per minute over the chart, and each learner’s score is their method’s true mean plus a personal amount equally likely to be -9, -3, 3 or 9. With n learners per group, how much does a group mean wander, and how often does the app group come out ahead?

Solution

The personal amounts average 0 and their mean square is (81 + 9 + 9 + 81)/4 = 45, so a single learner’s score wanders from its true mean with standard deviation \displaystyle \sqrt{45} \approx 6.71. Listing every equally likely combination of personal amounts for n learners gives the exact spread of a group mean, and comparing every app group with every chart group gives the chance the app comes out ahead (columns: learners per group, standard deviation of a group mean, chance the app group’s mean is higher):

per group	SD of mean	app ahead
1	6.71	0.625
2	4.74	0.742
5	3.00	0.837
10	2.12	0.903
20	1.50	0.972

The spread equals \displaystyle \sqrt{45/n}: going from 5 learners to 20, four times as many, halves it. The app’s true 4 never changes, but with one learner per method the chart learner wins more than a third of the time, and with 20 per method the app wins about 97 times in 100. The difference was always there; replication lets it show.

Example 4

Is it completely randomized?

Say whether each plan for Khoa’s 30 learners is a completely randomized design.

(a) Each learner rolls a die: 1 or 2, app; 3 or 4, metronome; 5 or 6, chart.

(b) The 15 morning learners are shuffled and split 5, 5, 5, and so are the 15 evening learners.

Solution

(a) Yes. Chance alone sends each learner, with no grouping first. The group sizes are left to chance too, so they will rarely be exactly 10 each; equal sizes are usual, not required.

(b) No. The learners are grouped by class first and randomized within each class. That is a randomized block design, a different design with its own purpose (see Boundaries).

(c) No, it is not random at all. Eager learners sign up first, and eagerness may help them practice, so it would be confounded with the app.

Example 5

One experiment is not the last word

Nhung repeats Khoa’s experiment at another school with 30 new learners, and Huy at a third. The app minus chart differences are 5.3 (Khoa), 4.6 (Nhung) and 6.1 (Huy). What does repeating add? And would filming one app learner, one metronome learner and one chart learner ten times each replicate the experiment?

Solution

The three differences average (5.3 + 4.6 + 6.1)/3 = 16/3 \approx 5.33 changes per minute, and all three are positive and similar. A single experiment could have been lucky in its assignment, or could depend on Khoa’s teaching; three schools with three teachers agreeing makes both explanations much less likely. That is replication of the experiment.

Ten films of one learner are not replication. They show how much one learner’s count varies from minute to minute, not how much learners differ from one another, and with one learner per method any gap between methods could be the gap between three people. Replication counts units, not measurements.

Example 6

What the design ignores

After the study Huy points out that 12 of the 30 learners already played guitar, and that the assignment happened to put 6 of them in the app group, 4 in the metronome group and 2 in the chart group. Suppose guitar players score about 6 changes per minute higher on any method. How much of the app’s 5.3 advantage could guitar explain? Was the design at fault?

Solution

Guitar players make up 6/10 of the app group and 2/10 of the chart group, so guitar alone lifts the app mean by about 0.6 \times 6 = 3.6 and the chart mean by 0.2 \times 6 = 1.2. About 3.6 - 1.2 = 2.4 of the 5.3 could be guitar.

The design worked as designed. A completely randomized design ignores everything known about the units, guitar included, and leaves its balance to chance. An even 4, 4, 4 split is not even likely: only about 0.11 of all assignments give it. The cost shows up as noise. Guitar is part of why scores within a group spread so widely, which made the chance gap of Example 2 as large as 1.80. Had Khoa known about guitar beforehand, he could have balanced it on purpose.

Boundaries

It uses no knowledge of the units. A variable known to affect the response (guitar experience, class time, the side of a field) is left to chance, and its variation swells the noise. Grouping alike units first and randomizing within each group is blocking.

Replication counts units. Measuring one unit many times, or treating all units in one shared container, does not give independent units: if a whole group’s units share one oven, tank or room, the experimental unit is that container.

More units shrink chance, not bias. Larger groups make the means steadier but cannot repair a measurer who knows the groups, a control that differs in more than the treatment, or dropouts.

It answers “did it cause?” for these units. Who the result applies to depends on how the units were chosen (Scope of inference: random selection and random assignment).

Where this goes next

Blocking: randomized block and matched pairs designs keeps the random assignment but first groups units that are alike, including matched pairs. Comparing two treatments by re-randomizing turns the question “is this difference bigger than chance?” into a computation, and the chance model of Example 3 is the seed of Sampling variability and simulating it and, later, the two-sample tests.

Exercises

Writing out a design

Huy wants to compare two new nutrient mixes for hydroponic spinach with the standard mix. He has 45 jars, each holding one plant and its own nutrient water, and will measure each plant’s height after four weeks. Write out a completely randomized design: units, treatments, assignment, response and comparison.
Describe the outline diagram for the design in Exercise 1: what each box says and where the arrows go.
A design compares 4 treatments with 8 units each. How many units does it need, and how many replicates does each treatment have?
In Exercise 1, why should all 45 jars be assigned in one random process rather than the standard mix going to the 15 jars nearest the window?

Computing with replication

Huy runs a smaller version with 6 jars per mix. Heights in centimeters: Mix A 18, 21, 17, 20, 22, 19; Mix B 16, 19, 15, 18, 17, 20; standard 15, 17, 14, 18, 16, 16. Find each mean and each new mix’s difference from the standard.
In the chance model of Example 3, find the standard deviation of a group mean with 45 learners per group, and with 80.
In the same model, what is the smallest group size that makes the standard deviation of a group mean 2 or less?
In the same model, the difference of two group means has standard deviation \displaystyle \sqrt{45}\sqrt{\tfrac{1}{n} + \tfrac{1}{n}}. Find it for 10 learners per group.
Change the model so the app truly adds 8 changes per minute, with one learner per method. List the 16 equally likely pairs of personal amounts and find the chance the app learner scores higher.
In a larger run of Huy’s study, 15 jars per mix, the plants within a mix vary with a pooled s \approx 3 cm. Mix A beats the standard by 2.6 cm and Mix B by 0.8 cm. Find the typical chance gap \displaystyle s\sqrt{\tfrac{1}{15} + \tfrac{1}{15}} and judge each difference.
Suppose the three repeats in Example 5 had given 5.3, -0.5 and 1.2. Find the mean and say what the repeats now suggest.
In a different random assignment, the 12 guitar players land 3 in the app group, 5 in the metronome group and 4 in the chart group. With the same 6-change lift, how much does guitar shift the app minus chart difference?

Is it completely randomized?

Each jar in Exercise 1 gets a card drawn from a well-mixed pile of 15 “A”, 15 “B” and 15 “standard” cards.
Jars on the top shelf are shuffled among the three mixes, and so are jars on the bottom shelf.
Huy gives each mix to the jars whose plants look most alike at the start.

Reasoning

Nhung brews kombucha at three temperatures. She has 18 jars, assigns 6 to each temperature at random, and puts each temperature’s 6 jars together in one warming cabinet. What is the experimental unit, and how much replication does the study really have?
Example 2 already had 10 learners per method. Explain why repeating the whole experiment still adds something.
A friend says a completely randomized design is pointless because it might put most guitar players in one group. Answer the friend, and say when the friend has a fair point.
Explain why a real difference between two treatments is easier to detect with 20 units per treatment than with 5, though the difference itself is the same.

Answers

Units: the 45 jars (one plant each, own water). Treatments: Mix A, Mix B and the standard mix (the control). Assignment: label the jars 01 to 45, shuffle once, first 15 Mix A, next 15 Mix B, last 15 standard. Response: plant height after four weeks, with light and temperature the same for all. Comparison: mean heights, each new mix against the standard.
A box “45 jars, the units” leads along a split marked “random assignment” to three boxes, “15 jars, Mix A”, “15 jars, Mix B”, “15 jars, standard (control)”; lines from all three join at a box “compare mean height after 4 weeks”.
4 \times 8 = 32 units; each treatment has 8 replicates.
Window jars get more light, which affects growth, so light would be confounded with the standard mix. One random process for all 45 lets nothing but chance decide.
Mix A 117/6 = 19.5 cm, Mix B 105/6 = 17.5 cm, standard 96/6 = 16 cm. Mix A beats the standard by 3.5 cm, Mix B by 1.5 cm.
\displaystyle \sqrt{45/45} = 1 and \displaystyle \sqrt{45/80} = 0.75 changes per minute.
\displaystyle \sqrt{45/n} \le 2 means n \ge 45/4 = 11.25, so 12 learners per group.
\displaystyle \sqrt{45}\sqrt{0.2} = \sqrt{9} = 3 changes per minute.
13/16 = 0.8125. The app learner trails only if the chart learner’s amount exceeds the app learner’s by more than 8: as (app, chart) amounts, (-9, 3), (-9, 9), (-3, 9), 3 of the 16.
\displaystyle 3\sqrt{2/15} \approx 1.10 cm. Mix A’s 2.6 is about 2.4 gaps, larger than chance usually makes; Mix B’s 0.8 is about 0.73 of a gap, an ordinary chance difference.
(5.3 - 0.5 + 1.2)/3 = 2. The repeats disagree, so Khoa’s 5.3 may have been partly luck or something about his school; the app’s advantage is in doubt until it is understood.
App lift 0.3 \times 6 = 1.8, chart lift 0.4 \times 6 = 2.4: the difference shifts by 1.8 - 2.4 = -0.6, against the app.
Completely randomized: one chance process, no grouping first, 15 per mix.
Not completely randomized: the jars are grouped by shelf first. It is a block design.
Not random: a person chooses, so nothing guarantees the groups differ only by chance.
The unit is the cabinet, since temperature is applied to a cabinet, and all 6 jars share whatever else that cabinet does. There is one unit per temperature: no real replication.
One run can be unlucky in its assignment or depend on its teacher, school or learners; a repeat with new units and people checks that the result holds beyond one setting.
Chance imbalance is possible but is already part of the chance variation the analysis allows for, so the comparison stays fair. The friend has a point when guitar is known beforehand to matter: blocking on it removes that noise.
The difference stays fixed while the chance spread of each group mean shrinks like \displaystyle 1/\sqrt{n}, half as large at 20 as at 5, so the difference stands out further from the noise.