Statistics

Slope from r and the spreads, and regression to the mean

If all you know about two variables is their means, their standard deviations and the correlation r, can you write down the least-squares line? And why are its predictions always closer to the mean than the values they come from?

You need

The least squares criterion and its lineStatistics The correlation coefficientAlgebra 1

Where you are 5 in Least squares regression

Statistics7

Bivariate data9

Least squares regression5

Interpreting the slope and intercept Predicting with a regression line, and extrapolation The least squares criterion and its line

Slope from r and the spreads, and regression to the mean

Fitting linear and exponential models with technology

When you reach for it

Narek, Gayane and Ashot weigh and measure the tortoises at a sanctuary. Their report keeps only summaries (made-up data):

	mean	standard deviation
shell length x (cm)	\bar{x} = 30	s_x = 4
mass y (kg)	\bar{y} = 9	s_y = 2.5

and the correlation r = 0.85. The raw pairs are gone. The least-squares line has slope b = S_{xy}/S_{xx} and passes through the point of averages (\bar{x}, \bar{y}), but S_{xy} and S_{xx} are built from the raw pairs. This lesson rewrites the slope in terms of the five summaries, so a line can be written from a report, a table in a paper, or a calculator’s summary statistics.

The rewrite also answers a second question. Measured in standard deviations, every prediction the line makes is closer to the mean than the value it was made from. That is regression to the mean, and the formula shows exactly where it comes from.

The procedure

The slope. The correlation is the sum of the products of z-scores divided by n - 1. Writing the z-scores out,

\begin{aligned} r &= \frac{1}{n-1}\sum \frac{x_i - \bar{x}}{s_x}\cdot\frac{y_i - \bar{y}}{s_y} \\ &= \frac{S_{xy}}{(n-1)\,s_x s_y}, \end{aligned}

so S_{xy} = (n-1)\,r\,s_x s_y. The standard deviation of x satisfies s_x^2 = S_{xx}/(n-1), so S_{xx} = (n-1)\,s_x^2. Divide the first by the second; the n - 1 and one s_x cancel:

b = \frac{S_{xy}}{S_{xx}} = \frac{(n-1)\,r\,s_x s_y}{(n-1)\,s_x^2} = r\,\frac{s_y}{s_x}.

The intercept. The line passes through (\bar{x}, \bar{y}), so \bar{y} = a + b\bar{x}, which gives a = \bar{y} - b\bar{x}. The slope comes first because the intercept is built from it.

Each part of b = r\,s_y/s_x has a job. Standard deviations are positive, so b has the sign of r. The number r has no units; the ratio s_y/s_x carries them, here kilograms per centimeter, which are the units a slope of mass against length must have.

The same line in z-scores. Write the line through the point of averages as \hat{y} - \bar{y} = b(x - \bar{x}), put in b = r\,s_y/s_x, and divide both sides by s_y:

\frac{\hat{y} - \bar{y}}{s_y} = r\,\frac{x - \bar{x}}{s_x}, \quad\text{that is,}\quad \hat{z}_y = r\,z_x .

In words: for each standard deviation that x sits above its mean, the line predicts y to sit r standard deviations above its mean. In particular, for standardized data the least-squares line is \hat{z}_y = r z_x, with slope r and no intercept.

Regression to the mean. Since -1 \le r \le 1 (Properties of the correlation coefficient),

|\hat{z}_y| = |r|\,|z_x| \le |z_x|,

with equality only when z_x = 0 or |r| = 1. Unless the points lie on one line, |r| < 1, and every prediction is fewer standard deviations from \bar{y} than x is from \bar{x}. A tortoise 2 standard deviations longer than average is predicted to be only 0.85 \times 2 = 1.7 standard deviations heavier than average.

This is arithmetic, not a force. Nothing pulls a tortoise toward the middle, and the masses keep their spread s_y. The line predicts the average mass of tortoises with a given length. Among the longest tortoises some are very heavy and some only fairly heavy, because the link is imperfect, so their average mass is less extreme than their length.

The figure sets the line \hat{z}_y = 0.85\,z_x beside the SD line z_y = z_x, the line through the point of averages that rises one standard deviation of y for each one of x (slope s_y/s_x in the original units). For a positive r the SD line runs along the long axis of the cloud; the least-squares line is flatter by the factor r.

Figure 1The sanctuary tortoises in z-units (made-up data). A shell 2 standard deviations above average predicts a mass only 1.7 standard deviations above average; the least-squares line is the SD line flattened by the factor r = 0.85.

Swapping the roles. The least squares criterion and its line showed that the line of x on y is not the line of y on x solved for x; the summaries now say by how much. To predict length from mass, the same steps with the letters traded give \hat{z}_x = r\,z_y, so the least-squares line of x on y has slope r\,s_x/s_y. That is not 1/b = s_x/(r\,s_y), the slope got by solving \hat{y} = a + bx for x. The two least-squares slopes multiply to

r\,\frac{s_y}{s_x}\cdot r\,\frac{s_x}{s_y} = r^2,

which is 1 only when |r| = 1. Drawn with x across, the line of x on y is z_y = z_x/r, steeper than the SD line while the line of y on x is flatter: each line regresses toward the mean of the variable it predicts.

Worked examples

Example 1

The tortoise line from five numbers

Write the least-squares line for predicting mass from shell length for the sanctuary tortoises, interpret its slope, and predict the mass of a tortoise with a 34 cm shell.

Solution

The slope is

b = r\,\frac{s_y}{s_x} = 0.85 \times \frac{2.5}{4} = 0.53125 \text{ kg per cm}.

The line passes through (30, 9), so

a = \bar{y} - b\bar{x} = 9 - 0.53125(30) = -6.9375,

and the line is \hat{y} = -6.9375 + 0.53125x. Each extra centimeter of shell predicts about 0.53 kg more mass (Interpreting the slope and intercept). The intercept only places the line: no tortoise has a shell anywhere near 0 cm.

At x = 34, \hat{y} = -6.9375 + 18.0625 = 11.125, so about 11.1 kg.

Example 2

Predicting in standard deviations

Use \hat{z}_y = r z_x to predict the mass of a sanctuary tortoise with a 38 cm shell and of one with a 24 cm shell. Check both with the line from Example 1.

Solution

A 38 cm shell has z_x = (38 - 30)/4 = 2, so \hat{z}_y = 0.85(2) = 1.7 and

\hat{y} = \bar{y} + 1.7\,s_y = 9 + 1.7(2.5) = 13.25 \text{ kg}.

A 24 cm shell has z_x = (24 - 30)/4 = -1.5, so \hat{z}_y = 0.85(-1.5) = -1.275 and \hat{y} = 9 - 1.275(2.5) = 5.8125 kg.

Check with the line: -6.9375 + 0.53125(38) = 13.25 and -6.9375 + 0.53125(24) = 5.8125. Both predictions are closer to the mean, in standard deviations, than the shells they came from: 1.7 against 2 above, 1.275 against 1.5 below. The pull toward the mean works from either side.

Example 3

Working back to r

Ashot’s report on a second sanctuary lost its correlation. It gives \bar{x} = 25 cm, s_x = 5 cm, \bar{y} = 6 kg, s_y = 2 kg and least-squares slope b = 0.32 kg per cm. Find r and the line.

Solution

Solve b = r\,s_y/s_x for r:

r = b\,\frac{s_x}{s_y} = 0.32 \times \frac{5}{2} = 0.8 .

It is positive, as it must be with a positive slope, and between -1 and 1, as it must be for any data. The intercept is a = 6 - 0.32(25) = 6 - 8 = -2, so the line is \hat{y} = -2 + 0.32x.

Example 4

Regression to the mean in a race

Gayane times sprinters in two heats of a 100-meter race (made-up data). In both heats the mean time is 13.0 s and the standard deviation 0.5 s, and the correlation between a sprinter’s two times is r = 0.6. Predict the second-heat time of a sprinter who ran 12.0 s in the first heat, and of one who ran 13.75 s. Narek says the fast ones slowed down because they relaxed after a good first heat. What does the arithmetic say?

Solution

Because the two heats have the same mean and the same standard deviation, the slope is r\,s_y/s_x = 0.6, and the z-form is the quickest route. The first sprinter has z_x = (12.0 - 13.0)/0.5 = -2, so \hat{z}_y = 0.6(-2) = -1.2 and the predicted time is 13.0 - 1.2(0.5) = 12.4 s. The second has z_x = (13.75 - 13.0)/0.5 = 1.5, so \hat{z}_y = 0.9 and the predicted time is 13.0 + 0.9(0.5) = 13.45 s. The fast sprinter is predicted slower, the slow one faster.

No relaxing is needed to explain this. A correlation of 0.6 says the first heat only partly predicts the second, so sprinters who were extreme in the first heat are, as a group, less extreme in the second. Run the line the other way and the same thing happens: the line of first heat on second heat also has slope 0.6, so a sprinter who ran 12.0 s in the second heat is predicted to have run 12.4 s in the first. Relaxing after the first heat cannot change the first heat, so the drift needs no cause between the heats; |r| < 1 is enough to produce it. And the second-heat times were given the same standard deviation, 0.5 s, as the first: the predictions bunch toward the mean, the times themselves do not.

Example 5

Predicting length from mass

The keepers find a sanctuary tortoise of 13.25 kg whose shell was not measured. Find the least-squares line for predicting shell length from mass, and use it. Compare with solving the line of Example 1 for x.

Solution

Mass is now the explanatory variable, so the slope is

r\,\frac{s_x}{s_y} = 0.85 \times \frac{4}{2.5} = 1.36 \text{ cm per kg},

and since this line also passes through the point of averages, its intercept is 30 - 1.36(9) = 17.76. The line is \hat{x} = 17.76 + 1.36y, and at y = 13.25 it gives 17.76 + 18.02 = 35.78 cm. In z-scores: z_y = (13.25 - 9)/2.5 = 1.7, so \hat{z}_x = 0.85(1.7) = 1.445 and \hat{x} = 30 + 1.445(4) = 35.78 cm, the same.

Solving 13.25 = -6.9375 + 0.53125x instead gives x = 38, the shell Example 2 started from. That inverted line has slope 1/0.53125 = \tfrac{32}{17} \approx 1.88 cm per kg, not 1.36, and the two least-squares slopes multiply to 0.53125 \times 1.36 = 0.7225 = 0.85^2, which is r^2. Inverting skips the regression: it answers 2 standard deviations of length for a mass only 1.7 standard deviations out. The line built to predict length regresses toward the mean length, to 1.445.

Boundaries

Both spreads must be positive. If s_x = 0 every x is the same and no line is determined. If s_y = 0 then r is undefined, and the least-squares line is the flat line \hat{y} = \bar{y}.
Both standard deviations must be computed the same way. The ratio s_y/s_x is the same with divisor n - 1 or n, as long as both use the same one.
Five numbers fix the line, not whether a line belongs. Curved data and data with an outlier can have exactly these summaries; only a scatter plot or a residual plot shows whether a straight line suits them.
Regression to the mean is about predictions, which are averages. It does not say every extreme individual moves toward the mean, and it does not shrink the spread of the data. When |r| = 1 there is none; when r = 0 it is total, and every prediction is \bar{y}.
The line predicts inside the data. A prediction for an x far from the observed ones is an extrapolation, whatever formula produced the line.
The line of x on y is a different line. To predict x from y, use slope r\,s_x/s_y, never 1/b, unless |r| = 1.

Where this goes next

Putting b = r\,s_y/s_x into the least-squares line’s leftover sum of squared residuals, S_{yy} - S_{xy}^2/S_{xx}, gives (1 - r^2)\,S_{yy}: the fraction of the spread in y the line leaves unexplained is 1 - r^2. That is r squared, in r squared and the standard deviation of the residuals. Software reports a, b, r squared and the residual spread in one table (Reading regression output). Since r and both standard deviations are pulled by a single far point, so is the slope (Outliers, leverage and influential points).

Exercises

Exercises 1, 2, 9 and 10 use Ashot’s island tortoises (made-up data): shell length \bar{x} = 26 cm, s_x = 5 cm; mass \bar{y} = 7 kg, s_y = 2 kg; r = 0.9.

Lines from summary statistics

Find the least-squares line for predicting mass from shell length on Ashot’s island, and interpret its slope.
Predict the mass of an island tortoise with a 31 cm shell twice: from the line of Exercise 1, and from \hat{z}_y = r z_x.
For the sanctuary’s tortoise eggs, incubation temperature x has mean 30 °C and standard deviation 1.5 °C, days to hatching y has mean 90 days and standard deviation 12 days, and r = -0.8. Find the least-squares line and interpret its slope. Does its intercept mean anything?
Gayane records the sanctuary tortoises’ masses in grams instead of kilograms. Give the new \bar{y}, s_y, r, slope and intercept.
A data set has r = 0.6, s_x = 10 and least-squares slope b = 0.3. Find s_y.
Both variables of a data set are standardized, and r = -0.6. Write the least-squares line and predict z_y when z_x = -2.

Regression to the mean

In Exercises 7 and 8, Narek times the sanctuary tortoises’ walking speeds on two days (made-up data). Day 1 speeds have mean 20 meters per hour and standard deviation 4; day 2 speeds have mean 22 and standard deviation 4; the correlation between the two days is r = 0.5.

Predict the day 2 speed of a tortoise that walked 28 meters per hour on day 1, and of one that walked 14 meters per hour.
A tortoise walked 30 meters per hour on day 2. Predict its day 1 speed. Why does this show that the effect in Exercise 7 needs no cause between the two days?
On Ashot’s island, predict the mass of a tortoise with a 36 cm shell from the SD line and from the least-squares line. Which prediction is closer to \bar{y}, and why must it be?

Swapping the variables

On Ashot’s island, find the least-squares line for predicting shell length from mass. Predict the shell length of an 8.8 kg tortoise from it, and again by solving the line of Exercise 1 for x.
For one data set the least-squares line of y on x has slope 0.5 and the line of x on y has slope 1.28. Find r.

Explaining

Starting from b = S_{xy}/S_{xx} and \displaystyle r = \frac{1}{n-1}\sum z_x z_y, show that b = r\,s_y/s_x.
Explain why the least-squares slope always has the sign of r, and why b has units while r has none.
What does \hat{z}_y = r z_x predict when r = 1, and when r = 0? Describe regression to the mean in each case.
Gayane finds that hatchlings of the heaviest mother tortoises are, in standard deviations, less far above the average hatchling mass than their mothers are above the average mother. She concludes that tortoise masses bunch together more each generation. Explain what is wrong.
Show that the slope of y on x times the slope of x on y is r^2, and use it to say when the line of x on y is the line of y on x solved for x.

Answers

b = 0.9 \times 2/5 = 0.36 kg per cm and a = 7 - 0.36(26) = -2.36, so \hat{y} = -2.36 + 0.36x. Each extra centimeter of shell predicts 0.36 kg more mass.
Both give 8.8 kg. The line: -2.36 + 0.36(31) = 8.8. The z-form: z_x = (31 - 26)/5 = 1, so \hat{z}_y = 0.9 and \hat{y} = 7 + 0.9(2) = 8.8.
b = -0.8 \times 12/1.5 = -6.4 days per °C and a = 90 + 6.4(30) = 282, so \hat{y} = 282 - 6.4x. Each degree warmer predicts 6.4 fewer days to hatching. The intercept is the prediction at 0 °C, far outside the data, so it has no meaning; it only places the line.
\bar{y} = 9000 g and s_y = 2500 g; r = 0.85, unchanged, since it has no units. The slope is 0.85 \times 2500/4 = 531.25 g per cm and the intercept 9000 - 531.25(30) = -6937.5 g: both are 1000 times their values in kilograms.
s_y = b\,s_x/r = 0.3 \times 10/0.6 = 5.
\hat{z}_y = -0.6\,z_x, a line through the origin. At z_x = -2, \hat{z}_y = 1.2.
28: z_x = 2, \hat{z}_y = 1, so 22 + 4 = 26 meters per hour. 14: z_x = -1.5, \hat{z}_y = -0.75, so 22 - 3 = 19 meters per hour.
z_y = (30 - 22)/4 = 2, so \hat{z}_x = 0.5(2) = 1 and the predicted day 1 speed is 20 + 4 = 24 meters per hour. The prediction moves toward the mean in both directions of time, and nothing on day 2 can change day 1; |r| < 1 alone produces the effect.
z_x = (36 - 26)/5 = 2. The SD line gives 7 + 2(2) = 11 kg; the least-squares line gives -2.36 + 0.36(36) = 10.6 kg, which is \hat{z}_y = 1.8. The least-squares prediction is closer to \bar{y}, because its slope is r = 0.9 times the SD line’s slope and |r| < 1.
Slope 0.9 \times 5/2 = 2.25 cm per kg and intercept 26 - 2.25(7) = 10.25, so \hat{x} = 10.25 + 2.25y. At y = 8.8: \hat{x} = 30.05 cm. Solving 8.8 = -2.36 + 0.36x gives x = 31 cm, the shell of Exercise 2; the line for length regresses back toward 26 cm.
r^2 = 0.5 \times 1.28 = 0.64, and r = 0.8, positive because both slopes are positive.
Writing out the z-scores, r = S_{xy}/\bigl((n-1)s_x s_y\bigr), so S_{xy} = (n-1)\,r\,s_x s_y. Also S_{xx} = (n-1)s_x^2. Dividing, b = r\,s_x s_y/s_x^2 = r\,s_y/s_x.
b = r\,s_y/s_x and s_y/s_x > 0, so b and r share a sign. A z-score has no units, so r, an average of their products, has none; s_y/s_x has units of y per unit of x, and b inherits them.
r = 1: \hat{z}_y = z_x, so every prediction is exactly as extreme as x; there is no regression, and every point lies on the line. r = 0: \hat{z}_y = 0 for every x, so the prediction is always \bar{y}; regression to the mean is total.
With |r| < 1, the line predicts hatchlings of extreme mothers to be less extreme, in standard deviations: arithmetic about averages, not a pull. It says nothing about the spread of hatchling masses, which can be as large as the mothers’. Run backward, the mothers of the heaviest hatchlings are also predicted to be less extreme, which would “show” masses spreading apart, the opposite conclusion from the same data.
r\,\dfrac{s_y}{s_x}\cdot r\,\dfrac{s_x}{s_y} = r^2. Solving the line of y on x for x gives slope 1/b. That equals the slope of x on y exactly when b times the slope of x on y is 1, that is, when r^2 = 1. Both lines pass through (\bar{x}, \bar{y}), so equal slopes make them the same line: only when the points lie exactly on a line.