Statistics

Conditions for regression inference

A sample’s slope comes with a standard error and a t distribution. What has to be true of the data before those numbers can be believed?

You need

Making and reading residual plotsAlgebra 1 Sampling distribution of the slopeStatistics

Unlocks

Interval for a slopeStatistics

Where you are 4 in Inference for slopes

Statistics7

Statistical inference8

Inference for slopes4

Sampling distribution of the slope

Conditions for regression inference

Interval for a slope Test for a slope

Why this exists

Pinar studies tea gardens near Rize, on the Black Sea coast of Turkey. For a random sample of plots she records the nitrogen fertilizer each plot received, x kilograms per hectare, and its leaf yield, y kilograms per plot, and fits the least-squares line \hat{y} = a + bx. Her slope b is a statistic; what she wants is \beta, the slope for every plot the estate could grow.

Sampling distribution of the slope described how b varies from sample to sample: centered on \beta, with standard deviation \displaystyle \sigma/(s_x\sqrt{n - 1}), normal in shape. Each of those facts was proved from a model of the population. If the model is wrong for Pinar’s gardens, the standard error and the t distribution built on it describe some other population, not hers. This lesson names the conditions that model requires and shows how each is checked on the data, mostly with a residual plot. It plays the part that Conditions for one mean plays for a mean.

Definition

The regression model. For each value of x the responses have mean \mu_y = \alpha + \beta x and standard deviation \sigma about that line; \alpha, \beta and \sigma are parameters. The sample estimates them with a, b and s, the standard deviation of the residuals y - \hat{y}.

The conditions for inference about a slope are five, and their first letters spell LINER.

Linear. The true mean of y is a straight-line function of x. Checked by a scatter plot that shows no curve and a residual plot with no curved pattern.
Independent. The observations are independent of one another. When cases are sampled without replacement, the sample is at most 10\% of the population (the The ten percent condition).
Normal. At each x the responses vary normally about the line. Checked by a histogram or dot plot of the residuals that shows no strong skew and no outliers.
Equal spread. The standard deviation \sigma is the same at every x. Checked by a residual plot whose vertical spread does not fan out or narrow.
Random. The data come from a random sample or a randomized experiment (Random sampling, Random assignment and confounding).

Intuition

Figure 1The regression model (made-up population). At every amount of nitrogen the yields are normal, centered on the line \mu_y = \alpha + \beta x, with the same standard deviation \sigma.

The model is one picture. Above every amount of nitrogen there is a whole population of plots. Their yields form a normal curve, the curve’s center sits on the line, and every curve has the same width. Each condition is one feature of the picture: the centers lie on a straight line (Linear), the curves are normal (Normal), the curves have one width (Equal spread), and the plots in the sample were picked by chance (Random), each without affecting the others (Independent).

We never see the population, only the sample. A residual y - \hat{y} is the sample’s stand-in for a plot’s miss from the true line, so the residuals are where the conditions show. A curve in the residuals says the centers do not lie on a line. A fan says the curves change width. A lopsided dot plot says the curves are not normal.

Results

> What each condition protects. > > | Condition | Fails, then | > |---|---| > | Linear | \beta is not a single rate of change; b and \hat{y} are biased at some x | > | Independent | \displaystyle \sigma/(s_x\sqrt{n - 1}) is the wrong spread for b | > | Normal | the t distribution with n - 2 degrees of freedom is not exact, badly so when n is small | > | Equal spread | the single s in \displaystyle SE_b = \dfrac{s}{s_x\sqrt{n - 1}} misstates the spread of b | > | Random | the results describe only the cases measured, and say nothing about cause |

> When a condition fails. Do not compute the interval or test as if it held. Either re-express > a variable and check the conditions again on the new fit > (Transforming to linearity: logs and exponential models), or report the line as a description of these data > together with the condition that failed.

Why it is true

In Sampling distribution of the slope the slope was written as

b = \beta + \sum w_i\,\varepsilon_i, \qquad w_i = \frac{x_i - \bar{x}}{\sum (x_j - \bar{x})^2},

where \varepsilon_i = y_i - (\alpha + \beta x_i) is plot i’s miss from the true line. Every step of that argument used a condition.

Linear. The center \mu_b = \beta needs each \varepsilon_i to have mean 0 at its own x. If the true means bend away from any line, the misses have means that change with x, and b settles on the tilt of whatever line best fits the bend, which depends on where the x values happen to be.
Independent and Equal spread. The spread came from adding variances, \sum w_i^2\sigma^2 = \sigma^2/\sum(x_i - \bar{x})^2. Adding variances is allowed only for independent misses, and pulling out one \sigma^2 is allowed only when every miss has the same \sigma.
Normal. A weighted sum of independent normal misses is exactly normal; with s in place of \sigma, (b - \beta)/SE_b then has exactly a t distribution with n - 2 degrees of freedom. Without normal misses this is only approximate, and the approximation improves as n grows.
Random. Chance selection is what makes the \varepsilon_i random misses from a population at all. Without it there is no sampling distribution to speak of.

Worked examples

Example 1

All five conditions met

Pinar takes a random sample of 15 plots from an estate of 400. Nitrogen x (kilograms per hectare) and leaf yield y (kilograms per plot), made-up data:

(60, 38.0), (80, 36.4), (100, 40.5), (120, 44.2), (140, 42.1), (160, 42.0), (180, 48.1), (200, 48.4), (220, 48.1), (240, 53.5), (260, 53.7), (280, 52.0), (300, 55.4), (320, 59.4), (340, 58.3).

Fit the line and check each condition.

Solution

Least squares gives

\hat{y} = 32.16 + 0.0792x, \qquad s = 1.82 \text{ kilograms}.

Each extra kilogram per hectare of nitrogen goes with about 0.079 more kilograms of leaf per plot. In order of nitrogen the residuals are 1.08, -2.10, 0.41, 2.53, -1.15, -2.84, 1.68, 0.39, -1.49, 2.32, 0.94, -2.34, -0.53, 1.89, -0.80.

Figure 2Pinar’s 15 plots (made-up data) with \hat{y} = 32.16 + 0.0792x, and their residual plot. No curve, no fan.

Linear: the scatter plot is a straight band and the residuals show no curve; positive and negative residuals are mixed along the whole range. Equal spread: the residuals reach about \pm 3 kilograms at low, middle and high nitrogen alike; no fan.

Figure 3Pinar’s residuals, each rounded to the nearest 0.5 kilogram. Roughly symmetric, no outliers.

Normal: the dot plot of residuals is roughly symmetric about 0, with no long tail and no dot far from the others. With 15 points it cannot prove normality, and it does not need to; it shows nothing a normal model would struggle to produce (Checking whether data look normal). Random: the plots were a random sample. Independent: sampling was without replacement, and 15 \le 0.10 \times 400 = 40. All five hold, so Pinar may go on to an interval or test for \beta.

Example 2

A fan

Gizem samples 12 plots at random on a large estate, made-up data: (40, 29.5), (60, 30.3), (80, 32.6), (100, 36.1), (120, 35.5), (140, 40.4), (160, 43.4), (180, 40.9), (200, 41.7), (220, 50.0), (240, 53.6), (260, 46.6). Check the equal spread condition and say what it means for her.

Solution

The fit is \hat{y} = 25.19 + 0.0991x with s = 2.77 kilograms.

Figure 4Gizem’s residuals (made-up data) fan out as nitrogen grows. The equal spread condition fails.

The residual plot fans out. The six residuals at nitrogen 40 to 140 are 0.35, -0.83, -0.51, 1.00, -1.58, 1.34, with standard deviation 1.13; the six at 160 to 260 are 2.36, -2.12, -3.30, 3.01, 4.63, -4.35, with standard deviation 3.75, more than three times as large. One s = 2.77 overstates the scatter at low nitrogen and understates it at high nitrogen, and the SE_b built on it is not the spread of b. The equal spread condition fails.

No curve shows, so a straight line still describes the trend. Gizem can report \hat{y} = 25.19 + 0.0991x as a description of her 12 plots and say that scatter grows with nitrogen. A larger sample would not help: it would show the same fan more clearly. Taking the logarithm of yield often evens out a fan like this; she would then refit and check every condition again on the new residuals.

Example 3

A curve

Dilara applies nitrogen to 12 plots on a third estate, two at each rate, made-up data: (0, 20.4), (0, 19.7), (16, 27.5), (16, 28.6), (64, 36.3), (64, 35.6), (144, 43.4), (144, 44.5), (256, 52.4), (256, 51.8), (400, 59.7), (400, 60.5). Check the linear condition, and try regressing y on \displaystyle \sqrt{x}.

Solution

The straight line is \hat{y} = 26.49 + 0.0924x with s = 4.18. Its residuals, nitrogen rate by rate, are -6.1 and -6.8 at 0; -0.5 and 0.6 at 16; 3.9 and 3.2 at 64; 3.6 and 4.7 at 144; 2.3 and 1.7 at 256; -3.7 and -2.9 at 400. Negative, then positive, then negative: an arch.

Figure 5Dilara’s residuals (made-up data). Above, from the straight line, an arch, so the linear condition fails. Below, from \displaystyle \hat{y} = 20.00 + 2.00\sqrt{x}, no pattern.

The yield rises fast at first and then levels off, and a line cannot follow that. The linear condition fails, and with it the meaning of \beta: there is no single rate of change to estimate.

Yield looks like a straight-line function of \displaystyle \sqrt{x} instead. With \displaystyle u = \sqrt{x}, the values 0, 4, 8, 12, 16, 20, least squares gives

\hat{y} = 20.00 + 2.00\sqrt{x}, \qquad s = 0.47.

Its residuals run from -0.64 to 0.58 with no arch and no fan, and s fell from 4.18 to 0.47. The conditions are now checked again on this fit, and any interval or test is for the slope in the \displaystyle \sqrt{x} scale: 2.00 more kilograms per unit of \displaystyle \sqrt{x}. For example, 100 kilograms per hectare predicts 20.00 + 2.00(10) = 40.0 kilograms per plot.

Example 4

Independent and Random in three designs

Decide whether the independent and random conditions hold.

(a) Gizem picks 15 plots at random from a small estate of 120. (b) Gizem randomly assigns five nitrogen rates to 15 plots of her own garden, three plots each. (c) Dilara weighs the harvest of the same 3 plots on 5 dates, for 15 observations.

Solution

(a) Random holds. Independence fails the 10% check: 0.10 \times 120 = 12 and 15 > 12. Drawing a large share of a small estate without replacement makes each plot’s chance of selection depend on which plots were already drawn, and the standard error formula does not account for that.

(b) Random holds through random assignment. The 10% check does not apply, because no plots were drawn from a larger population; the chance comes from the assignment. Because the rates were assigned at random, a real slope can be read as nitrogen causing the change in yield, for plots like those in her garden.

(c) Independence fails. Five harvests of one plot share its soil, slope and tea bushes, so they move together; the data hold 3 plots, not 15 independent ones. Treating them as 15 would make SE_b far too small.

Example 5

An outlier in the residuals

A hailstorm strikes one more of Pinar’s sampled plots, which gave (200, 38.0). Add it to her data and compare.

Solution

From her line, the residual is 38.0 - (32.16 + 0.0792 \times 200) = -10.0, which is 10.0/1.82 \approx 5.5 residual standard deviations below the line. On the dot plot it would stand far to the left of every other residual. The normal condition fails.

Refitting all 16 plots gives \hat{y} = 31.54 + 0.0792x with s = 3.13. The slope did not move, because the new plot sits at \bar{x} = 200 and so pulls the line straight down without tilting it (Outliers, leverage and influential points). But s grew from 1.82 to 3.13, and the standard error grew with it. The x values still have \sum(x - \bar{x})^2 = 112{,}000, so

\begin{aligned} SE_b &= \frac{1.82}{\sqrt{112{,}000}} \approx 0.00544 \text{ without it}, \\ SE_b &= \frac{3.13}{\sqrt{112{,}000}} \approx 0.00935 \text{ with it}. \end{aligned}

One plot changed the precision of the slope by a factor of about 1.7. The hail is a cause outside the question about nitrogen, so Pinar reports the analysis without that plot and says why, or reports both; she does not delete it silently.

Boundaries

The conditions are about the population. The data can show that one clearly fails; they can never prove that one holds. “Met” means no plot showed a clear problem.
Check Linear first. If the trend curves, the residuals are measured from the wrong line, and their spread and shape say little about the other conditions.
Nothing is assumed about the shape of the x values. The Normal condition is about the responses about the line, so a histogram of x or of y alone does not check it.
The Normal condition matters less as n grows. Linear, Independent and Equal spread do not ease with sample size, and a fan does not shrink.
The conditions cover the range of x observed. Nothing in the data checks the model beyond it.
Mild imperfections are expected; a residual plot of real data is never perfectly even. A condition fails when the pattern is clear, not when it is imaginable.

Where this goes next

With the conditions checked, (b - \beta)/SE_b has a t distribution with n - 2 degrees of freedom. Interval for a slope turns that into an interval b \pm t^{*}SE_b for \beta, and Test for a slope tests whether \beta = 0. Both read b, s and SE_b from Reading regression output. When a condition fails, re-expression is the usual repair: Transforming to linearity: logs and exponential models, including Log-log transforms and power models.

Exercises

All data are made up.

Checking conditions with numbers

Pinar’s line is \hat{y} = 32.16 + 0.0792x with s = 1.82. A plot outside her sample had x = 150 and y = 47.3. Find its residual and how many residual standard deviations it is.
A random sample of 20 plots is drawn without replacement. Is the 10% condition met if the estate has 180 plots? 250 plots?
What is the smallest estate from which 15 plots can be sampled without replacement and meet the 10% condition?
In a residual plot, the four residuals at the lowest nitrogen are 0.5, -0.4, 0.3, -0.4 and the four at the highest are 3.1, -2.6, 2.2, -2.7. Find the standard deviation of each group. Which condition is in doubt?
A regression has s = 1.8 kilograms, and one plot has residual -6.3. How many residual standard deviations is that, and what does it suggest?
A typing error in one yield gave s = 3.0; once corrected, s = 2.0, with the same x values. By what factor does SE_b change?
In order of nitrogen, a regression’s residuals are -2.1, -0.8, 0.9, 1.7, 2.2, 1.4, 0.3, -1.2, -2.4. Which condition does this suggest fails?

Which condition, and what to do

For each study, name the condition in doubt and say what to do.

Dilara weighs leaf from one plot every week for 15 weeks, raising its nitrogen each week.
Gizem’s 15 plots are the 15 nearest the estate road.
A random sample of 10 plots gives a residual dot plot with a long tail to the right.
A random sample of 60 plots gives a residual plot with no curve and no fan, and a residual histogram that is slightly skewed.
A scatter plot of yield against nitrogen rises steeply and then flattens.
A random sample of 40 plots gives residuals that spread wider at high nitrogen.

Explaining

Explain why a residual plot can show a curve that the scatter plot hides.
Explain why the 10% condition plays no part in Gizem’s experiment in Example 4(b).
Explain why “the residual plot fans out, but I will report the interval and mention it” is not an acceptable analysis.
In Example 5 the hail plot changed s but not b. Explain why a point at x = \bar{x} cannot change the slope.
Explain why the Normal condition eases as n grows but the Equal spread condition does not.
Pinar’s dot plot of residuals looks roughly symmetric. Why does she still look at the residual plot before going on?

Answers

Residual 3.26, about 1.8 residual standard deviations. \hat{y} = 32.16 + 11.88 = 44.04, 47.3 - 44.04 = 3.26, and 3.26/1.82 \approx 1.8: an ordinary plot.
Not met for 180 plots, since 0.10 \times 180 = 18 and 20 > 18. Met for 250, since 0.10 \times 250 = 25 \ge 20.
150 plots. Need 15 \le 0.10N, so N \ge 150.
About 0.47 and 3.08, a ratio of about 6.6. Both groups have mean 0; for example \displaystyle \sqrt{(0.25 + 0.16 + 0.09 + 0.16)/3} \approx 0.47. The spread grows with nitrogen: Equal spread.
-6.3/1.8 = -3.5. A residual that far out is an outlier: the Normal condition is in doubt, and the plot’s record should be checked for a cause.
It is multiplied by 2/3. \displaystyle SE_b = s/(s_x\sqrt{n - 1}) and only s changed.
Linear. The signs run negative, then positive five times, then negative: an arch, so the true mean curves.
Independent fails: weekly yields of one plot share soil and weather and carry over from week to week. There is also only one plot. Use many plots, each with its own rate.
Random fails: roadside plots may differ from the rest. Report the line for these plots only.
Normal: with only 10 plots a strong skew matters. Try re-expressing yield and check again, or report the line without an interval or test.
All met as described, provided the estate has at least 600 plots so that 60 \le 0.10N. A slight skew is no problem with 60 plots, since b is close to normal.
Linear fails. Re-express x (a square root or logarithm) and check the residuals of the new fit.
Equal spread fails, and 40 plots do not fix it. Try the logarithm of yield and check again, or report the line descriptively.
A residual plot subtracts the line and stretches the vertical scale to the size of the misses, so a bend that is small beside the overall trend becomes plain.
No plots were sampled from a larger population. The chance comes from random assignment, and the 10% condition is about sampling without replacement.
The standard error rests on one \sigma for every x. With a fan, SE_b is not the spread of b, so the stated confidence level or P-value is not the true one; the numbers would claim a precision they do not have. Mentioning it does not repair it.
Because \sum(x - \bar{x}) = 0, the slope b = \sum(x - \bar{x})(y - \bar{y})/\sum(x - \bar{x})^2 equals \sum(x - \bar{x})\,y/\sum(x - \bar{x})^2, so \bar{y} plays no part. A point with x = \bar{x} leaves \bar{x} unchanged and adds 0 to both sums, so b is the same; only the intercept and s move.
b is a weighted sum of many misses, so it becomes nearly normal as n grows whatever their shape. A fan is a feature of the population; more data estimate it more sharply but do not remove it.
The dot plot shows only the shape of the residuals. A curve or a fan is about how residuals change with x, which a dot plot throws away.