Statistics

Sampling distribution of the slope

Ten test plots gave a fitted slope. Ten other plots would have given a different one. How far can a sample’s slope be from the true slope of the whole population?

You need

Least squares regressionGrades 11 to 12 Sampling distributionsGrades 11 to 12

Unlocks

Conditions for regression inferenceStatistics

Where you are 4 in Inference for slopes

Statistics7

Statistical inference8

Inference for slopes4

Sampling distribution of the slope

Conditions for regression inference Interval for a slope Test for a slope

Why this exists

Nahuel runs ten quinoa test plots in the Andes. For each plot he records the rainfall over the growing season, x millimeters, and the yield, y kilograms, and fits the least-squares line \hat{y} = a + bx. His slope is b = 0.0096: about 0.0096 more kilograms per plot for each extra millimeter of rain.

That number describes ten plots. What Nahuel wants to know is how yield depends on rain for all the plots that could be grown this way. Ten other plots would have given another slope; that is Sampling variability and simulating it again, now for a slope instead of a mean or a proportion. Before a sample slope can be trusted, we need to know how much slopes wander from sample to sample, and around what. That pattern is the sampling distribution of the slope, and it is what makes an interval or a test for a slope possible.

Definition

The population regression model. For each value of x there is a whole population of responses. Their mean lies on a straight line,

\mu_y = \alpha + \beta x,

and individual responses scatter above and below that line with the same standard deviation \sigma at every x. Here \alpha and \beta are parameters: fixed, usually unknown numbers about the population. In this lesson the made-up population of quinoa plots has

\mu_y = 2.0 + 0.010x, \qquad \sigma = 0.6 \text{ kilograms}.

A plot that gets 300 millimeters of rain has mean yield 2.0 + 0.010(300) = 5.0 kilograms, and single plots at 300 millimeters typically miss that by about 0.6 kilograms.

The sample line. A sample of n cases gives the least-squares line \hat{y} = a + bx. Its intercept a estimates \alpha, its slope b estimates \beta, and the standard deviation of its residuals, s, estimates \sigma. The slope b is a statistic: computed from the sample, and different for each sample.

The sampling distribution of the slope is the distribution of b over all possible samples of the same size, taken the same way, from the same population. Like any of the Sampling distributions, it has a center, a spread and a shape. Its mean is written \mu_b and its standard deviation \sigma_b.

Intuition

Take the made-up population above and draw samples from it. Every sample here uses the same ten rainfall amounts, in millimeters:

160,\ 240,\ 240,\ 260,\ 280,\ 320,\ 340,\ 360,\ 360,\ 440,

which have mean 300 and standard deviation s_x = 80. Only the yields are drawn fresh, each one the population mean at its rainfall plus a random miss with standard deviation 0.6. Each sample then gets its own least-squares line.

Figure 1Five made-up samples of ten plots from one made-up population. Thick line, the population line \mu_y = 2.0 + 0.010x. Dots and solid thin line, Nahuel’s sample and its least-squares line. Dashed lines, the least-squares lines of four more samples.

The five sample lines all tilt upward like the population line, but not by the same amount: their slopes are 0.0096, 0.0060, 0.0106, 0.0133 and 0.0098. They cross near the middle of the rainfall values, because every least-squares line passes through its own (\bar{x}, \bar{y}), and \bar{y} changes little from sample to sample. The wobble is in the tilt.

Now repeat the sampling 500 times and draw a histogram of the 500 slopes.

Figure 2The slopes of 500 simulated samples of ten plots (made-up population). They center on the true slope, and the normal curve with standard deviation 0.0025 fits them.

Three things show. The slopes pile up around the true slope 0.010, not to one side of it. They spread out by about 0.0025 on either side. And the pile is a single symmetric mound that a normal curve fits.

Why should the spread of the rainfall amounts matter? Think of a seesaw. If all ten plots had rainfall close to 300 millimeters, a little random noise at the left end and a little at the right end could tip the line a long way, because the two ends are close together. Plots spread far apart act like a long seesaw: the same noise in the yields changes the tilt much less.

Results

> The sampling distribution of b. Suppose the population follows \mu_y = \alpha + \beta x > with standard deviation \sigma about the line, the observations are independent, and samples of > n have x values with standard deviation s_x. Then > > - Center: \mu_b = \beta. The slope b is an Bias and variability of an estimator of \beta. > - Spread: \displaystyle \sigma_b = \dfrac{\sigma}{s_x\sqrt{n - 1}}. > - Shape: exactly normal when the responses are normal about the line; approximately normal > for large n otherwise. > > The spread is smaller with more data (larger n), less scatter about the line (smaller \sigma), > and x values spread farther apart (larger s_x).

> The standard error of the slope. In practice \sigma is unknown, so the residual standard > deviation s is put in its place. The result, \displaystyle SE_b = \dfrac{s}{s_x\sqrt{n - 1}}, estimates > \sigma_b from one sample. This is the Standard error that regression software prints in the \text{SE Coef} column, in > the row of the explanatory variable (see Reading regression output).

Why it is true

Treat the x values as fixed and write S_{xx} = \sum (x_i - \bar{x})^2, which equals (n - 1)s_x^2 by the definition of s_x. The slope from r and the standard deviations, b = r\dfrac{s_y}{s_x}, becomes after writing out r

\begin{aligned} b &= \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{S_{xx}} \\ &= \frac{\sum (x_i - \bar{x})\,y_i}{S_{xx}}, \end{aligned}

since \sum (x_i - \bar{x}) = 0 makes the \bar{y} part vanish. Each response is y_i = \alpha + \beta x_i + \varepsilon_i, where the miss \varepsilon_i has mean 0 and standard deviation \sigma. Using \sum (x_i - \bar{x}) = 0 again, \sum (x_i - \bar{x})(\alpha + \beta x_i) = \beta \sum (x_i - \bar{x})x_i = \beta S_{xx}, so

b = \beta + \sum w_i\,\varepsilon_i, \qquad w_i = \frac{x_i - \bar{x}}{S_{xx}}.

Center. Every \varepsilon_i has mean 0, so the mean of b is \beta.

Spread. The misses are independent, so variances add: the variance of b is \sigma^2 \sum w_i^2 = \sigma^2 \dfrac{S_{xx}}{S_{xx}^2} = \dfrac{\sigma^2}{S_{xx}}. Taking the square root, \displaystyle \sigma_b = \dfrac{\sigma}{\sqrt{S_{xx}}} = \dfrac{\sigma}{s_x\sqrt{n - 1}}.

Shape. b is \beta plus a weighted sum of the misses. A weighted sum of independent normal quantities is normal. If the misses are not normal, a sum of many small independent pieces is still close to normal, in the same spirit as the central limit theorem, as long as no single x value carries most of the weight.

Worked examples

Example 1

Center and spread for Nahuel’s design

Nahuel’s samples have n = 10 plots with s_x = 80 millimeters, drawn from the population \mu_y = 2.0 + 0.010x with \sigma = 0.6 kilograms. Describe the sampling distribution of b.

Solution

The center is the true slope, \mu_b = \beta = 0.010 kilograms per millimeter, because b is unbiased. The spread is

\sigma_b = \frac{0.6}{80\sqrt{10 - 1}} = \frac{0.6}{80 \times 3} = 0.0025,

in kilograms per millimeter. The units of \sigma_b are the units of the slope. On a scale people feel, the true slope is 1 kilogram per 100 millimeters, and a sample slope typically misses that by about 0.25 kilograms per 100 millimeters. With the responses normal, the distribution is normal, so about 95\% of samples give a slope within 2\sigma_b = 0.005 of 0.010: between 0.005 and 0.015.

Example 2

Reading the simulation

The 500 simulated slopes in the histogram have mean 0.01005 and standard deviation 0.00250; 479 of them lie between 0.005 and 0.015, and 334 between 0.0075 and 0.0125. Compare with Example 1, and place Nahuel’s own slope, b = 0.0096, in the distribution. His ten plots were (160, 2.87), (240, 4.63), (240, 5.00), (260, 4.29), (280, 4.00), (320, 5.16), (340, 5.69), (360, 6.26), (360, 4.87), (440, 5.62).

Solution

The simulation agrees with the formulas. Its mean 0.01005 is only 0.00005 above \beta = 0.010; 500 samples cannot hit the center exactly, but there is no lean to one side. Its standard deviation 0.00250 matches \sigma_b = 0.0025. The share within two standard deviations is 479/500 = 95.8\% and within one is 334/500 = 66.8\%, close to the 95\% and 68\% a normal curve gives.

Nahuel’s slope stands

z = \frac{0.0096 - 0.010}{0.0025} = -0.16

standard deviations below the center: an ordinary sample. His slope misses \beta by 0.0004, small beside the typical miss of 0.0025. In real work Nahuel never sees the histogram or \beta; the simulation shows what his one b is a draw from.

Example 3

More plots

Rocio plans 40 plots instead of 10, with the same rainfall spread s_x = 80 millimeters. Find \sigma_b and compare with Nahuel’s design.

Solution

\sigma_b = \frac{0.6}{80\sqrt{39}} \approx \frac{0.6}{499.6} \approx 0.0012.

Four times as many plots cut \sigma_b from 0.0025 to 0.0012, a factor of \displaystyle \sqrt{9}/\sqrt{39} \approx 0.48, a little better than half. The n sits under a square root, so the standard deviation falls slowly: halving it costs about four times the data.

Example 4

Spreading the rainfall

Ayelen can choose where her ten plots go. Sites in one valley give rainfall with s_x = 40 millimeters; sites from the dry foothills to the wet slopes give s_x = 120. Find \sigma_b for each. How many plots in the one valley would match Nahuel’s \sigma_b = 0.0025?

Solution

With n = 10, \displaystyle \sqrt{n - 1} = 3:

\begin{aligned} s_x = 40&: \quad \sigma_b = \frac{0.6}{40 \times 3} = 0.005, \\ s_x = 120&: \quad \sigma_b = \frac{0.6}{120 \times 3} \approx 0.0017. \end{aligned}

Tripling the spread of x divides \sigma_b by 3, for free: the same ten plots, placed farther apart. To reach 0.0025 in the one valley she needs \displaystyle \dfrac{0.6}{40\sqrt{n - 1}} = 0.0025, so \displaystyle \sqrt{n - 1} = \dfrac{0.6}{40 \times 0.0025} = 6, n - 1 = 36 and n = 37 plots. Spreading the x values out did the work of 27 extra plots.

Example 5

The standard error from one sample

Rocio’s own sample is 12 plots with mean rainfall \bar{x} = 290 millimeters, s_x = 95, and a least-squares slope b = 0.0094 with residual standard deviation s = 0.52 kilograms. Find SE_b, and explain the software line for her data (made-up printout):

Predictor	Coef	SE Coef
Constant	2.41	0.50
Rainfall	0.0094	0.00165

with \text{S} = 0.52.

Solution

She does not know \sigma, so s takes its place:

SE_b = \frac{0.52}{95\sqrt{11}} \approx \frac{0.52}{315.1} \approx 0.00165.

That is the \text{SE Coef} in the Rainfall row: an estimate, from her one sample, of how far a slope like hers typically lands from \beta. Had she known \sigma = 0.6, her design’s true value would be \sigma_b = 0.6/315.1 \approx 0.0019; her s came out a little below 0.6, so her estimate is a little low. A different sample would give a different s and a different SE_b. (The 0.50 beside the Constant is the matching standard error for the intercept, \displaystyle s\sqrt{\tfrac{1}{n} + \tfrac{\bar{x}^2}{S_{xx}}}, which this lesson does not need.)

Example 6

If rain made no difference

Suppose instead that rain had no effect on yield: \beta = 0, still with \sigma = 0.6, n = 10 and s_x = 80. How likely would a sample slope of 0.010 or more be?

Solution

The sampling distribution would be normal with center 0 and the same \sigma_b = 0.0025, since \sigma_b does not depend on \beta. A slope of 0.010 is

z = \frac{0.010 - 0}{0.0025} = 4

standard deviations above the center, and P(Z \ge 4) \approx 0.00003: about 3 in 100{,}000 samples. A slope that large would be very hard to get by chance from a population with no relationship. Turning this reasoning into a formal procedure, with SE_b in place of \sigma_b, is the test for a slope.

Boundaries

The formulas rest on the model: the mean of y really is linear in x, the observations are independent, \sigma is the same at every x, and the responses are normal about the line (or n is large). Checking these is Conditions for regression inference, using Making and reading residual plots. If the trend curves, b estimates the slope of a line that is not there.
When cases are drawn without replacement, independence needs the sample to be at most about 10\% of the population.
The x values are treated as fixed, and then \displaystyle \sigma_b = \sigma/(s_x\sqrt{n - 1}) is exact. When the x values also change from sample to sample, the results hold given the x values a sample actually got, with that sample’s own s_x; this is how SE_b is used. The center \beta holds either way, but there is then no single \sigma_b for all samples, because s_x changes too.
At least two different x values are needed: with s_x = 0 there is no slope at all. SE_b also needs n \ge 3, since s divides by n - 2.
Unbiased describes the center of all samples, not one sample. A single b can miss \beta by several \sigma_b.
A small \sigma_b says the slope is pinned down. It does not say that rain causes the yield, or that the line describes rainfall outside the range sampled.

Where this goes next

Knowing the center and spread of b turns one sample slope into a statement about \beta. Conditions for regression inference sets out when the model holds. Because \sigma is unknown and SE_b is used instead, (b - \beta)/SE_b follows a t distribution with n - 2 degrees of freedom; Interval for a slope builds an interval b \pm t^{*}SE_b from it, and Test for a slope asks, as Example 6 began to, whether a sample slope is too far from 0 to be chance. The background is Least squares regression and the general idea of Sampling distributions.

Exercises

Center and spread

In Exercises 1 to 4, all data are made up.

A population of plots follows \mu_y = 2.0 + 0.010x with \sigma = 0.6 kilograms. Samples have n = 17 plots with s_x = 75 millimeters. Give the center and standard deviation of the sampling distribution of b.
Same population, samples of n = 26 plots with s_x = 50 millimeters. Find \sigma_b.
A rougher growing region has \sigma = 0.9 kilograms about the same line. Find \sigma_b for n = 10 and s_x = 80.
Another crop follows \mu_y = 1.5 + 0.008x with \sigma = 0.4. Samples have 5 plots with s_x = 50. Give the center and standard deviation of b.

Using the normal shape

Exercises 5 to 7 use Nahuel’s design: b is normal with mean 0.010 and \sigma_b = 0.0025.

Find the probability that a sample slope lies between 0.008 and 0.012.
Find the probability that a sample slope is above 0.014.
Find the interval that holds the middle 90\% of sample slopes.
A tiny study of 5 plots has s_x = 40, from the population \mu_y = 2.0 + 0.010x with \sigma = 0.6. Find the probability that its slope is negative, pointing the wrong way.

What makes the spread smaller

With s_x = 80 and \sigma = 0.6, find \sigma_b for n = 50 plots.
A hardier seed cuts \sigma to 0.3 kilograms. Find \sigma_b for n = 10 and s_x = 80.
Rocio’s sites give s_x = 100. What is the smallest number of plots that makes \sigma_b \le 0.002, with \sigma = 0.6?
With s_x = 80 and \sigma = 0.6, what is the smallest n for \sigma_b \le 0.001?
Design A uses 20 plots with s_x = 50; design B uses 10 plots with s_x = 90. With \sigma = 0.6, which gives the smaller \sigma_b?

The standard error and the printout

Ayelen’s sample has n = 15, s = 0.48 kilograms and s_x = 70 millimeters. Find SE_b.
A printout for 11 plots shows Rainfall with Coef 0.0112 and SE Coef 0.0021, and \text{S} = 0.61. Find the standard deviation s_x of the rainfall amounts.
Another sample under Nahuel’s design (n = 10, s_x = 80) has s = 0.66. Find SE_b and compare it with \sigma_b = 0.0025.

Explaining

Explain why spreading the x values farther apart makes the slope less variable.
“b is unbiased, so my slope equals \beta.” What is right and what is wrong in this?
All ten of a sample’s plots had exactly 300 millimeters of rain. What can be said about b?
Explain why SE_b changes from sample to sample while \sigma_b does not.
The simulation is run with 5000 samples instead of 500. What changes in the histogram, and what stays the same?
Why do the sample lines in the first figure cross near the middle of the rainfall values?

Answers

Center 0.010; \sigma_b = \dfrac{0.6}{75 \times 4} = 0.002 kilograms per millimeter.
\sigma_b = \dfrac{0.6}{50 \times 5} = 0.0024.
\sigma_b = \dfrac{0.9}{80 \times 3} = 0.00375.
Center 0.008; \sigma_b = \dfrac{0.4}{50 \times 2} = 0.004.
0.576. The bounds are z = \pm 0.8, and P(-0.8 < Z < 0.8) \approx 0.576.
0.0548. z = (0.014 - 0.010)/0.0025 = 1.6, and P(Z > 1.6) \approx 0.0548.
0.0059 to 0.0141: 0.010 \pm 1.645 \times 0.0025 \approx 0.010 \pm 0.0041.
0.0912. Here \sigma_b = \dfrac{0.6}{40 \times 2} = 0.0075, z = -0.010/0.0075 = -4/3, and P(Z < -4/3) \approx 0.0912. Nearly one tiny study in ten would point the wrong way.
\sigma_b = \dfrac{0.6}{80 \times 7} \approx 0.00107.
\sigma_b = \dfrac{0.3}{80 \times 3} = 0.00125, half of 0.0025: \sigma_b is proportional to \sigma.
10 plots. Need \displaystyle \sqrt{n - 1} \ge \dfrac{0.6}{100 \times 0.002} = 3, so n - 1 \ge 9.
n = 58. Need \displaystyle \sqrt{n - 1} \ge \dfrac{0.6}{80 \times 0.001} = 7.5, so n - 1 \ge 56.25, and n - 1 must be the whole number 57.
Design B. A: \displaystyle \dfrac{0.6}{50\sqrt{19}} \approx 0.00275; B: \dfrac{0.6}{90 \times 3} \approx 0.00222. Fewer plots, spread wider, win here.
\displaystyle SE_b = \dfrac{0.48}{70\sqrt{14}} \approx 0.00183.
About 92 millimeters. From \displaystyle SE_b = \dfrac{s}{s_x\sqrt{n - 1}}, \displaystyle s_x = \dfrac{0.61}{0.0021\sqrt{10}} \approx 91.9.
SE_b = \dfrac{0.66}{80 \times 3} = 0.00275, above 0.0025 because this sample’s s came out above \sigma = 0.6. Both are 10\% high: SE_b is off by the same proportion as s.
The slope is rise over run between the ends of the data. The same random misses in y change the rise by about the same amount, so a longer run turns them into a smaller change of slope. In the formula, s_x is in the denominator.
Right: over all samples, slopes average out to \beta. Wrong: one slope is a single draw and typically misses \beta by about \sigma_b.
There is no least-squares slope: every line through (300, \bar{y}) has the same squared misses, and \displaystyle \sigma_b = \sigma/(s_x\sqrt{n - 1}) would divide by 0. The sample says nothing about how rain changes yield.
\sigma_b is built from \sigma, a fixed fact about the population. SE_b uses s, computed from the sample, which changes with the sample.
The histogram gets smoother and its mean and standard deviation land closer to 0.010 and 0.0025. The center, spread and shape it is estimating stay the same: they depend on n, s_x and \sigma, not on how many samples are simulated.
Every least-squares line passes through (\bar{x}, \bar{y}). All samples share \bar{x} = 300, and \bar{y} varies only a little, so the lines pivot near that point and differ mostly in tilt.