Algebra 2 · S.IC.B.5

Comparing two treatments by re-randomizing

The group that got the treatment did better. Could the luck of who landed in which group have done that on its own?

You need

Simulating a sampling distribution and a margin of errorAlgebra 2 Observational studies and experimentsAlgebra 1

Where you are 7 in Experimental design

Statistics7

Sampling and study design6

Experimental design7

Comparing two treatments by re-randomizing

Control groups, placebos and blinding Units, treatments, factors and levels Random assignment and confounding Completely randomized design and replication ▾ Show all 7 (2 more)

When you reach for it

A randomized experiment ends with two groups and two summaries, and the summaries differ. The group given the treatment recalled more words, or lasted longer, or left a tip more often. Experiments with random assignment can show cause, but only once one doubt is out of the way: the groups were formed by chance, and chance alone never splits a set of subjects perfectly evenly. Some difference would appear even if the treatment did nothing at all.

So the question is not “do the groups differ?” They always do. It is: is this difference bigger than the luck of the assignment would usually produce? Re-randomizing answers it by doing the assignment again, many times, on the same results, and watching how big the differences come out when the treatment is given no part to play. It is the same move as simulating a sampling distribution, with one change: there the chance was in which members were sampled, here it is in which subjects were put in which group.

Reach for it whenever an experiment used random assignment to two treatments and you want to judge a difference in means or in proportions between the groups. All the experiments and numbers in this lesson are made up.

The procedure

Say what “no effect” would mean. Suppose the treatment makes no difference at all. Then every subject would have produced exactly the same result whichever group chance had put them in. The 20 numbers recorded would be the same 20 numbers; only the labels on them, “treatment” and “control”, came from the shuffle. This supposition, that the treatment does nothing, is the model being tested. The rest of the procedure asks what the data would look like if it were true.

Re-do the random assignment on the same results. Write each subject’s result on a card, ignore the old labels, shuffle, and deal the cards into two piles of the same sizes as the real groups. Under the model, this deal is just as likely as the one that really happened, because the real one was made by the same kind of shuffle. Compute the same statistic as before: the difference in means, \bar{x}_{\text{treatment}} - \bar{x}_{\text{control}}, or the difference in proportions. That is one re-randomized difference.

Repeat, many times. One deal is one dot. A few hundred or a thousand deals show the pile that chance assignment makes, which is why a computer does the dealing. Two groups of 10 can be dealt in 184\,756 different ways, far too many to list by hand; a thousand random deals give a faithful picture of all of them. For very small experiments every deal can be listed, and Example 5 does that.

Look at where the pile sits. It centers near 0, because under the model neither pile is favored: a deal and the same deal with the piles swapped are equally likely, and their differences are opposites. Its spread shows how big a difference chance assignment alone tends to produce for these results.

Place the real difference against the pile. Count the re-randomized differences that are at least as large as the one observed, and divide by the number of deals. If the experiment set out to test whether the treatment raises the result, count the deals at or above the observed difference. If it set out to ask whether the treatment changes the result in either direction, count the deals at least as far from 0 on either side. That direction is settled before the data is seen, because choosing it after looking would let the data pick its own test.

Read the share. It answers one question: if the treatment did nothing, how often would the shuffle alone give a difference this large?

A small share means chance assignment rarely does this. Either an unusual deal happened, or the treatment made a difference. The smaller the share, the harder the first explanation is to believe, and since the groups were formed at random, the treatment is then the reasonable explanation for the difference.
A large share means differences like this one are ordinary under the model. The data give no evidence of an effect. That is not evidence of no effect: a small experiment can miss a real but modest one.
In between, the evidence is in between. There is no line where weak evidence turns into proof; the honest report gives the share, the size of the difference and the size of the experiment, and lets the reader weigh them. In later work this share is called a p-value.

Report it as one simulation’s result. A second run of 1000 deals gives a slightly different count. The share from a run is an estimate of the share over all possible deals, and more deals make the estimate sharper without changing what it estimates.

Worked examples

Example 1

One re-randomization by hand

Priyanka wants to know whether a short walk helps people remember. She recruits 20 volunteers and shuffles cards with their names to put 10 in each group. The walking group spends 10 minutes walking around a playing field; the sitting group spends the same 10 minutes sitting quietly. Then everyone studies the same list of 30 words for 2 minutes and writes down as many as they can. The number of words each person recalled:

Group	Words recalled
walk	14, 17, 15, 18, 16, 17, 13, 16, 19, 15
sit	12, 15, 13, 11, 14, 16, 12, 13, 15, 14

Find the observed difference in means. Then carry out one re-randomization, in which the shuffled cards put these 10 results in the “walk” pile: 15, 15, 14, 14, 16, 17, 13, 15, 18, 15.

Solution

The walking group recalled 160 words in all and the sitting group 135, so

\begin{aligned} \bar{x}_{\text{walk}} - \bar{x}_{\text{sit}} &= \frac{160}{10} - \frac{135}{10} \\ &= 16.0 - 13.5 = 2.5 \end{aligned}

The walkers recalled 2.5 more words on average.

For the re-randomization, all 20 results go into one pool, whose total is 160 + 135 = 295. The dealt “walk” pile adds to

\begin{aligned} &15 + 15 + 14 + 14 + 16 \\ &\quad + 17 + 13 + 15 + 18 + 15 = 152 \end{aligned}

so its mean is 15.2. The “sit” pile holds the other 10 cards. Every deal splits the same pool, so their total is 295 - 152 = 143 and their mean is 14.3. This deal’s difference is

15.2 - 14.3 = 0.9

Nothing about walking entered this number: it is what one chance split of these 20 people produced. One deal says little. The next example deals a thousand times.

Example 2

A thousand re-randomizations

A computer re-dealt Priyanka’s 20 results into two piles of 10, 1000 times, and recorded each difference in means (Figure 1). The table counts the deals in the two tails.

Figure 1One computer run of 1000 re-randomizations of the walking experiment. Each dot is the difference in means from one random re-dealing of the same 20 results. The observed difference, 2.5, sits far out in the right tail (shaded).

Difference d	Deals at d	Deals at -d
1.5	19	23
1.7	13	20
1.9	5	12
2.1	9	13
2.3	7	7
2.5	1	1
2.7	0	0
2.9	1	1

Priyanka set out to test whether walking improves recall. Describe the pile, find the share of deals at least as large as the observed 2.5, and say what the experiment shows.

Solution

The pile. The 1000 differences average -0.0094, essentially 0, as the model of no effect predicts. They form a single hump. The tails in the table hold 55 deals at 1.5 or more and 77 at -1.5 or less, so 1000 - 55 - 77 = 868 deals, most of them, gave a difference between -1.3 and 1.3. Each difference is a sum of whole numbers divided by 10, and since the pool total 295 is odd, every difference lands on an odd number of tenths: 0.1, 0.3, 0.5 and so on, which is why the table moves in steps of 0.2.

The share. The question was one-directional (does walking help?), so count the deals at 2.5 or above: 1 at 2.5, 0 at 2.7, 1 at 2.9.

\frac{1 + 0 + 1}{1000} = \frac{2}{1000} = 0.002

In this run, 2 deals in 1000 gave a difference as large as Priyanka’s.

The conclusion. If walking did nothing, a split as lopsided as hers would turn up about twice in a thousand random assignments. That is very hard to put down to luck, so the experiment gives strong evidence that the walk raised recall. Because chance, not the volunteers, decided who walked, the difference can be read as caused by the walk. It applies to these 20 volunteers; carrying it to people in general would need them to be a random sample of people in general, the question Scope of inference: random selection and random assignment takes up.

The figure 0.002 belongs to this run. A second run of 1000 deals might give 1 or 4; any run would show the observed 2.5 far out in the tail.

Example 3

A difference that is not surprising

A florist tests sachets of flower food. She takes 20 bunches of the same flowers, assigns 10 at random to vases with flower food and 10 to plain water, and counts the days until each bunch wilts.

Vase	Days
flower food	9, 7, 10, 8, 11, 8, 9, 7, 10, 9
plain water	8, 9, 7, 10, 8, 6, 9, 8, 10, 7

She wants to know whether the food changes vase life in either direction. A computer re-randomized the results 1000 times (Figure 2):

Figure 2One run of 1000 re-randomizations of the flower experiment. The shaded bars, at least 0.6 from zero on either side, hold 399 of the 1000 differences.

d	Deals	d	Deals
-1.8	1	0	136
-1.6	3	0.2	124
-1.4	7	0.4	109
-1.2	31	0.6	79
-1.0	34	0.8	56
-0.8	50	1.0	29
-0.6	77	1.2	18
-0.4	117	1.4	12
-0.2	115	1.6	2

What does the experiment show?

Solution

The totals are 88 days with food and 82 without, so

\begin{aligned} &\bar{x}_{\text{food}} - \bar{x}_{\text{water}} \\ &\quad = 8.8 - 8.2 = 0.6 \end{aligned}

The question allows either direction, so count the deals at least 0.6 from 0 on either side. On the right, 0.6 and above:

79 + 56 + 29 + 18 + 12 + 2 = 196

On the left, -0.6 and below:

77 + 50 + 34 + 31 + 7 + 3 + 1 = 203

That is 196 + 203 = 399 deals, a share of 0.399 in this run.

About two deals in five give a difference at least this large with the food doing nothing. A difference of 0.6 days is just what the luck of the assignment produces, so the experiment gives no evidence that the food changes vase life. It does not show that the food is useless either: with 10 bunches a side and results that vary by several days, a modest real effect would be hard to see. A larger experiment could settle more.

Example 4

A difference in proportions

Ikenna waits tables at a café and wants to know whether drawing a small smiley face on the bill makes customers more likely to leave a tip. He takes the next 40 bills of the evening and chooses 20 of them at random to get the drawing. Of the 20 bills with the drawing, 15 came back with a tip, and 9 of the 20 others did. A computer re-randomized the results 1000 times (Figure 3):

Figure 3One run of 1000 re-randomizations of the tipping experiment. The shaded bars, at 0.3 or more, hold 52 of the 1000 differences.

d	Deals	d	Deals
-0.5	1	0	259
-0.4	11	0.1	205
-0.3	36	0.2	95
-0.2	118	0.3	39
-0.1	223	0.4	13

Describe how the re-randomizing works here, find the share, and judge the result.

Solution

The observed proportions are \displaystyle \frac{15}{20} = 0.75 and \displaystyle \frac{9}{20} = 0.45, so the observed difference is

0.75 - 0.45 = 0.30

Each result is now “tip” or “no tip”. Under the model of no effect, each table would have tipped or not whichever group it was in, so the pool is 15 + 9 = 24 cards marked “tip” and 16 marked “no tip”. Each deal puts 20 cards in the “drawing” pile. If s of them are marked “tip”, the other pile holds 24 - s, and the difference is

\begin{aligned} \frac{s}{20} - \frac{24 - s}{20} &= \frac{2s - 24}{20} \\ &= \frac{s - 12}{10} \end{aligned}

so the possible differences go in steps of 0.1, and the real deal, s = 15, gives 0.3.

Ikenna asked whether the drawing helps, one direction, so count the deals at 0.3 or more:

\frac{39 + 13}{1000} = \frac{52}{1000} = 0.052

About one deal in twenty gives a difference this large with the drawing doing nothing. That is some evidence that the drawing helps, not strong evidence: an assignment this lopsided is uncommon but far from rare. Listing all 137\,846\,528\,820 possible deals, which a counting formula does without listing them, gives a share of about 0.053, so this run of 1000 landed close. The reasonable next step is the one a scientist would take, a larger experiment, rather than a verdict either way from this one.

Example 5

Every deal, listed

A cycle shop tries a sealant on six identical bicycle tires, three chosen at random to get it. It records how many days each tire lasts before it needs pumping: 12, 10 and 8 days with the sealant, 7, 6 and 5 without. Find the exact share of deals that give a difference at least as large as the observed one, and say what it shows about an experiment of this size.

Solution

The observed means are 10 and 6 days, a difference of 4.

With only six tires every deal can be listed. The number of ways to choose which 3 of the 6 get the “sealant” label is

\binom{6}{3} = 20

and all 20 are equally likely under the shuffle (combinations count them). The pool totals 48 days. If the sealant pile adds to S, the other adds to 48 - S, and

\begin{aligned} \bar{x}_{\text{sealant}} - \bar{x}_{\text{none}} &= \frac{S - (48 - S)}{3} \\ &= \frac{2S - 48}{3} \end{aligned}

This is at least 4 exactly when 2S - 48 \ge 12, that is, S \ge 30. The largest three numbers in the pool, 12 + 10 + 8, make exactly 30, and swapping any of them for a smaller one lowers the sum. So only the real deal reaches 4, and the exact share is

\frac{1}{20} = 0.05

This is the most lopsided result six tires can give, and still one deal in twenty produces it by chance. Had the shop asked about either direction, the mirror deal (sealant on 7, 6, 5) would count too, and the share would be \displaystyle \frac{2}{20} = 0.1. However well the sealant works, three tires a side can never give stronger evidence than this. Small experiments are limited not by the treatment but by how few ways the luck can fall.

Boundaries

The groups must have been formed by chance. Re-randomizing imitates the random assignment that really happened. In an observational study the subjects sorted themselves, so no shuffle produced the real groups, and a small share shows only that the difference is too big to be a chance split, not that the explanatory variable caused it. A confounding variable could explain it just as well.

It judges chance, not anything else. A small share rules out luck of the assignment as a likely explanation. It does not rule out a badly run experiment: if the walkers also got more encouragement, or the flowers with food stood in a cooler spot, the difference could come from that. Treating the groups alike in everything but the treatment is the job of the design, Completely randomized design and replication and control groups and blinding.

A small share is not a large effect. The share says how surprising the difference is, not how big or how useful. A huge experiment can make a tiny difference surprising. Report the difference itself alongside the share.

A large share is not proof of no effect. It says the data are consistent with no effect. With small groups or results that vary a lot, a real effect can sit inside the chance pile.

Small groups cap the evidence. With n subjects split into groups of k and n - k, the smallest possible one-direction share is 1 \big/ \binom{n}{k}, reached only when the most lopsided deal is the real one. For 3 and 3 that is 0.05.

The share depends on the question asked beforehand. Counting one tail or both, and comparing means or medians, are choices made before the data are seen. Chosen afterwards, they can be bent toward whatever answer is wanted.

The conclusion covers the subjects in the experiment. Random assignment supports cause for these subjects. Saying the same of a larger population needs the subjects to have been chosen at random from it.

A simulated share is an estimate. One run of 1000 deals gave 0.052 where the exact share is about 0.053. More deals bring the estimate closer; they never change the share being estimated.

Where this goes next

Re-randomizing is the plainest form of a Randomization tests, and the share it reports is a p-value, the central number of Hypothesis testing. Formulas later stand in for the shuffling: the pile of differences in means is approximated by the curve behind the two-sample t-test, and the pile of differences in proportions by the one behind the two-sample z-test. Those formulas are quicker, but they rest on conditions; the shuffle needs only that the assignment was random.

Experiments built in pairs or blocks re-randomize differently, shuffling within each pair rather than across the whole pool, which is the idea of blocking.

Exercises

One re-randomization

In a re-randomization of Priyanka’s experiment (Example 1), the “walk” pile gets 13, 16, 12, 17, 15, 14, 19, 11, 16, 15. Find this deal’s difference in means.
In a re-randomization of the tire experiment (Example 5), the “sealant” pile gets 12, 6 and 5. Find the difference in means.

Reading a re-randomization pile

Use the table in Example 2. Suppose the observed difference had been 2.1 and the question was whether walking helps. What share of the 1000 deals is at least as large?
Use the table in Example 2. Suppose Priyanka had asked whether walking changes recall in either direction. What share of the deals lies at least 2.5 from 0?
Use the table in Example 3. If the observed difference in vase life had been 1.2 days, with the question still about either direction, what would the share be? What would it say?
Use the table in Example 3. Find the share of deals at 0.6 or more, and explain why it is close to half of 0.399.

Proportions

Ikenna repeats the experiment and this time 16 of 20 bills with the drawing and 8 of 20 others come back with a tip. Find the observed difference. Explain why the pile of Example 4 still applies, and use it to find the share.
Fifty volunteers are assigned at random, 25 to a stretching routine and 25 to none, and 18 and 11 respectively report no sore muscles the next day. Find the observed difference in proportions and describe the cards for one re-randomization.
Use the table in Example 4. What share of the deals lies at least 0.2 from 0 on either side?

Listing every deal

In how many ways can 8 subjects be dealt into two groups of 4? Two groups of 10 from 20?
Six identical spinning tops, three chosen at random, get a drop of oil on the point. They spin for 9, 7 and 8 seconds with the oil and 6, 5 and 10 seconds without. Find the observed difference and the exact share of deals at least as large.
Eight hens, four chosen at random, get a new feed. In a two weeks the hens on the new feed lay 14, 12, 11 and 9 eggs, and those on the usual feed lay 10, 8, 7 and 6. Find the observed difference in means and the exact share of deals at least as large.
What is the smallest one-direction share any experiment with groups of 3 and 3 can give? With 4 and 4?

Reasoning

Explain why a pile of re-randomized differences always centers near 0.
A survey compares the recall of 10 people who say they walk every day with 10 who say they do not, and finds a difference of 2.5 words. Someone re-randomizes the 20 results and gets a share of 0.002. Does this show that walking improves recall? Explain.
A second computer run of 1000 deals for Priyanka’s experiment gives 4 deals at 2.5 or above, not 2. Does that contradict Example 2? What would 10000 deals change?
A classmate says: “Ikenna’s share was 0.052, which is above 0.05, so the drawing does nothing.” Explain what is wrong with this reasoning.
Explain why the share counts deals at least as large as the observed difference, rather than deals exactly equal to it.
Suppose Priyanka’s walking group had again beaten the sitting group by 2.5 words, but every person’s result had been far more spread out, from about 5 to about 25 words. Would the share be larger or smaller? Explain.
A share of 0.002 in Priyanka’s experiment and a share of 0.002 in an experiment with 5000 volunteers per group could come with very different differences in means. Explain why, and what should be reported besides the share.

Answers

0.1. The pile adds to 148, mean 14.8; the other pile adds to 295 - 148 = 147, mean 14.7; and 14.8 - 14.7 = 0.1.
About -0.67. The sealant pile has mean \displaystyle \frac{23}{3}, the other pile (10, 8, 7) has mean \displaystyle \frac{25}{3}, and the difference is \displaystyle -\frac{2}{3}.
0.018. At 2.1 or above there are 9 + 7 + 1 + 0 + 1 = 18 deals, and \displaystyle \frac{18}{1000} = 0.018.
0.004. Two deals at 2.5 or above and two at -2.5 or below, \displaystyle \frac{4}{1000} = 0.004.
0.074. At 1.2 or above: 18 + 12 + 2 = 32; at -1.2 or below: 31 + 7 + 3 + 1 = 42; so \displaystyle \frac{74}{1000} = 0.074. About 7 deals in 100 would give a difference this large with no effect: some evidence of an effect, but not strong.
0.196, from 196 deals. The pile is close to symmetric about 0, since a deal and its swap are equally likely, so the two tails hold nearly equal counts (196 and 203) and one tail is about half of both.
0.4, since \displaystyle \frac{16}{20} - \frac{8}{20} = 0.8 - 0.4 = 0.4. Again 24 bills came back with a tip and 16 without, so the pool of cards is the same and so are its deals. The share is \displaystyle \frac{13}{1000} = 0.013, stronger evidence than the first experiment.
0.28, since \displaystyle \frac{18}{25} - \frac{11}{25} = 0.72 - 0.44 = 0.28. The pool is 18 + 11 = 29 cards marked “no soreness” and 21 marked “sore”; shuffle all 50 and deal 25 to the stretching pile.
0.313. At 0.2 or above: 95 + 39 + 13 = 147; at -0.2 or below: 118 + 36 + 11 + 1 = 166; and \displaystyle \frac{313}{1000} = 0.313.
\binom{8}{4} = 70 and \binom{20}{10} = 184\,756.
Difference 1; share \displaystyle \frac{7}{20} = 0.35. The means are 8 and 7. The pool totals 45, so a difference of 1 or more needs the oiled pile to add to 24 or more. Of the 20 triples from 5, 6, 7, 8, 9, 10, seven do: 27, 26, 25, 24 (with 10 and 9), 25, 24 (with 10 and 8) and 9 + 8 + 7 = 24. A difference this size is ordinary.
Difference 3.75; share \displaystyle \frac{2}{70} = \frac{1}{35} \approx 0.029. The means are 11.5 and 7.75. The pool totals 77, so the new-feed pile must add to 46 or more, and only 14 + 12 + 11 + 10 = 47 and 14 + 12 + 11 + 9 = 46 do among the 70 deals.
\displaystyle \frac{1}{20} = 0.05 and \displaystyle \frac{1}{70} \approx 0.014. At best only the real deal is that extreme, one of \binom{6}{3} = 20 or \binom{8}{4} = 70 equally likely deals.
Under the shuffle, a deal and the same deal with the two piles swapped are equally likely (when the groups are the same size), and their differences are opposites, so positive and negative differences balance. With unequal groups the center is still 0, because each result is equally likely to land in either pile, so each pile’s mean averages the pool mean.
No. The groups chose themselves, so the people who walk daily may differ in other ways (age, sleep, health) that affect recall. The small share says the difference is too big to be a chance split, but the split was never made by chance; it cannot separate walking from those other variables.
No. Both are estimates of the same share over all possible deals, and counts as small as 2 or 4 in 1000 vary from run to run. Both say a difference of 2.5 is rare under no effect. Ten thousand deals would give a sharper estimate of that share; they would not change the share itself.
A share of 0.052 says that a difference this large comes from chance assignment about one time in twenty. That is moderate evidence that the drawing helps, not evidence that it does nothing, and 0.05 is not a line where evidence switches off: 0.048 and 0.052 say almost the same thing. A large share would not show that the drawing does nothing either, only that this experiment could not detect an effect.
Any single exact value is uncommon when there are many possible values, so “exactly 2.5” would be rare even for an ordinary difference. The question is whether the difference is this large or larger, so every deal at least as extreme counts as matching it.
Larger. With results that vary much more, the re-randomized piles also vary more, so the pile of differences is wider and many more deals reach 2.5 by chance. The same difference is then less surprising.
With huge groups the chance pile is very narrow, so even a small difference, perhaps a fraction of a word, can sit far out in its tail. The share measures surprise, not size. Report the difference itself, with the group sizes, alongside the share.