Statistics

Outliers, leverage and influential points

One point sits apart from the rest of a scatter plot. Does it matter to the fitted line, and how can you tell?

You need

Least squares regressionGrades 11 to 12 Outliers and the 1.5 IQR ruleMath 6

Where you are 9 in Bivariate data

Statistics7

Bivariate data9

▴ 2 above Least squares regression5 Line of best fit

Outliers, leverage and influential points

Reading regression output Residuals3 ▾ Show all 9 (2 more)

Why this exists

A least-squares line makes the sum of the squared residuals as small as possible (The least squares criterion and its line). Squaring makes a big miss very expensive, so the line will bend a long way toward a point it would otherwise miss badly. One unusual point can therefore decide the slope that is meant to describe all the others.

With one variable, an outlier is a value far from the rest of its list, and the Outliers and the 1.5 IQR rule marks it. With two variables a point can be unusual in two different directions: its y can sit far from the pattern of the other points, or its x can sit far from the other x values. These two kinds behave very differently, and only one of them tends to move the line. This lesson names the kinds and gives the test that tells them apart: remove the point and fit again.

Definition

Take a least-squares line \hat{y} = a + bx fitted to a set of points, with residuals y - \hat{y} (Residuals) and residual standard deviation s (Reading regression output).

An outlier in regression is a point whose y is far from the pattern of the other points: its residual is large. A common guideline flags a point whose residual is more than 2s away from 0.
A point has high leverage when its x is far from the mean \bar{x} of the other points’ x values. A z-score for x, z = \dfrac{x - \bar{x}}{s_x} computed from the other points, measures how far.
A point is influential when removing it and fitting the line again changes the slope, the intercept or the predictions noticeably. Removing a point and refitting is the test.

The three labels overlap. A point can be any one, two or all three of them, and the worked examples meet the main combinations.

Intuition

The least-squares line always passes through the center point (\bar{x}, \bar{y}). Picture it as a seesaw on that pivot. A point near the middle of the x values pushes close to the pivot: it can lift the whole board a little, by raising \bar{y}, but it has almost no grip to tilt it. A point far out in x pushes at the end of a long board. If it sits where the board already points, it holds the board in place and the line does not move. If it sits well off that direction, it tilts the board toward itself, and the slope changes. Leverage is the length of the board at that point; influence is what happens when a point with leverage disagrees with the others.

Results

The slope is a weighted average. For each point with x_i \ne \bar{x}, let m_i be the slope of the segment from the center (\bar{x}, \bar{y}) to that point:

m_i = \frac{y_i - \bar{y}}{x_i - \bar{x}}

Then the least-squares slope is the average of these slopes, each weighted by (x_i - \bar{x})^2:

b = \frac{\sum (x_i - \bar{x})^2\, m_i}{\sum (x_i - \bar{x})^2}

A point twice as far from \bar{x} carries four times the weight. A point at x = \bar{x} carries none.

A point added at the mean of x. If a point (\bar{x}, y_0) is added to n points whose line is \hat{y} = a + bx, the new slope is still b and the intercept rises by the point’s residual shared among all n + 1 points:

b_{\text{new}} = b, \qquad a_{\text{new}} = a + \frac{e}{n + 1}, \qquad e = y_0 - \bar{y}

Why it is true

The least-squares slope is

b = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}

which is the same number as b = r\,\dfrac{s_y}{s_x} from Slope from r and the spreads, and regression to the mean. For a point with x_i \ne \bar{x}, the top term is (x_i - \bar{x})(y_i - \bar{y}) = (x_i - \bar{x})^2 m_i, by the definition of m_i. A point with x_i = \bar{x} adds 0 to the top and to the bottom. So b is the sum of the weights times the m_i, divided by the sum of the weights: a weighted average.

Now add (\bar{x}, y_0). The mean of x does not change, because the new x equals it. The new point adds 0 to the bottom sum. The mean of y moves to \bar{y}' = \bar{y} + \dfrac{y_0 - \bar{y}}{n + 1}, and the top sum becomes

\begin{aligned} &\sum (x_i - \bar{x})(y_i - \bar{y}') \\ &\quad = \sum (x_i - \bar{x})(y_i - \bar{y}) - (\bar{y}' - \bar{y}) \sum (x_i - \bar{x}) \end{aligned}

plus 0 from the new point. The deviations x_i - \bar{x} add to 0, so the top sum is unchanged and b_{\text{new}} = b. The intercept is a = \bar{y} - b\bar{x}, so it rises by exactly \bar{y}' - \bar{y} = \dfrac{e}{n + 1}. Finally, e is the point’s residual from the old line, because that line predicts \bar{y} at \bar{x}.

Worked examples

Example 1

A herd far from the pattern, near the middle

Tigist recorded herd size x (bison) and grazing range y (square kilometers) for ten bison herds (made-up data): (20, 86), (28, 92), (35, 122), (42, 132), (50, 149), (55, 177), (63, 183), (70, 212), (78, 221), (89, 252). Technology gives \hat{y} = 30.73 + 2.488x, with r = 0.994 and s = 6.52 km². For these herds \bar{x} = 53 and s_x = 22.32. Herd A, at (56, 240), is added, and the refitted line is \hat{y} = 34.82 + 2.531x, with r = 0.926 and s = 23.04 km². Is herd A an outlier? Does it have high leverage? Is it influential?

Solution

The ten-herd line predicts \hat{y} = 30.73 + 2.488(56) = 170.06 km² for a herd of 56 bison, so herd A’s residual is 240 - 170.06 = 69.94 km². The guideline’s cutoff is 2s = 13.04 km², and herd A’s residual is more than five times that. Herd A is an outlier: its range is about 70 km² more than herds of its size have.

Its size is close to the center: z = \dfrac{56 - 53}{22.32} \approx 0.13. Low leverage.

Refitting changed the slope from 2.488 to 2.531 km² per bison, a change of 0.043. At a herd of 80 bison the two lines predict 229.77 and 237.30 km², a rise of 7.53 km². The new line is the old one lifted a little, as the figure shows, and barely tilted. Herd A is an outlier but not influential on the line.

Figure 1Herd A (made-up data). Dashed, the line for the ten herds; solid, the line refitted with A. The line rises a little but hardly tilts.

The outlier does damage the measures of fit: r fell from 0.994 to 0.926 and s more than tripled. How one point moves r is taken up in Properties of the correlation coefficient.

Example 2

A herd far out in size, on the trend

A herd B of 180 bison with a range of 479 km² is added to Tigist’s ten herds instead. The refitted line is \hat{y} = 30.61 + 2.491x, with r = 0.999 and s = 6.15 km². Classify herd B.

Solution

Its size gives z = \dfrac{180 - 53}{22.32} \approx 5.69: far beyond every other herd, so herd B has high leverage. The ten-herd line predicts 30.73 + 2.488(180) = 478.57 km², so its residual is 479 - 478.57 = 0.43 km², well inside 2s = 13.04. It is not an outlier.

Refitting moved the slope from 2.488 to 2.491, a change of 0.003 km² per bison. Herd B sits where the seesaw already pointed, so it holds the line in place: high leverage, not influential. The correlation rose from 0.994 to 0.999, because a point far out along the line stretches the cloud lengthwise without adding scatter.

Figure 2Herd B (made-up data). Far out in herd size, but on the trend, so the refitted line (solid) barely moves and hides the dashed line for the ten herds.

Example 3

Removing a point and refitting

Dawit’s file holds Tigist’s ten herds and an eleventh, herd C: 180 bison kept inside a fenced preserve, with a range of 230 km². Fitted to all eleven, technology gives \hat{y} = 104.89 + 0.989x, with r = 0.762 and s = 38.72 km²; for the eleven sizes \bar{x} = 64.55 and s_x = 43.75. Does the 2s guideline flag herd C? Remove it, refit, and decide whether it is influential.

Solution

With all eleven herds, the line predicts 104.89 + 0.989(180) = 282.91 km² for herd C, a residual of 230 - 282.91 = -52.91 km². The cutoff is 2s = 77.44 km², so the guideline does not flag it. Even its z-score looks moderate: \dfrac{180 - 64.55}{43.75} \approx 2.64.

Both yardsticks were built with herd C inside them. It pulled the line toward itself, which shrank its own residual, and it widened both s and s_x. Measured against the other ten, its size has z \approx 5.69, as in Example 2.

Removing herd C leaves Tigist’s ten herds, whose line is \hat{y} = 30.73 + 2.488x. The slope more than doubles, from 0.989 to 2.488 km² per bison, and r rises from 0.762 to 0.994. From the line fitted without it, herd C’s residual is 230 - 478.57 = -248.57 km², about 38 times s = 6.52. For a herd of 120 bison the two lines predict 329.29 and 223.57 km², a gap of 105.72 km². Herd C is influential.

Figure 3Herd C (made-up data). One far herd off the trend drags the line flat.

What to report is a judgment the numbers cannot make. Herd C’s range is set by a fence, not by how far a herd of its size would roam, so it comes from a different situation. Fitting without it is reasonable if the report says so: the line describes free-ranging herds of 20 to 89 bison, and herd C is shown and discussed apart.

Example 4

Why herd C won

Use the weighted-average result to see why one herd out of eleven set the slope of Example 3’s eleven-herd line.

Solution

The eleven sizes add to 710, so \bar{x} = \dfrac{710}{11} \approx 64.55. Herd C sits 180 - \dfrac{710}{11} = \dfrac{1270}{11} \approx 115.45 bison from the center, so its weight is \left(\dfrac{1270}{11}\right)^2 \approx 13{,}329.75. The total of all eleven weights, \sum (x_i - \bar{x})^2, is 19{,}144.73, so herd C alone carries \dfrac{13{,}329.75}{19{,}144.73} \approx 0.696, about 70\% of the weight. By contrast, the herd of 63 bison sits \dfrac{17}{11} \approx 1.55 from the center and has weight \left(\dfrac{17}{11}\right)^2 \approx 2.39.

Herd C’s own slope from the center is m = \dfrac{230 - 168.73}{115.45} \approx 0.53, where \bar{y} = \dfrac{1856}{11} \approx 168.73. With 70\% of the weight on a slope near 0.53, the average is dragged far below the ten herds’ 2.488, down to 0.989.

Example 5

A herd added at the mean size

Bereket recorded five reintroduced bison herds: years since reintroduction x and herd size y, (1, 30), (2, 40), (3, 80), (4, 90), (5, 110) (made-up data). Their least-squares line is \hat{y} = 7 + 21x. A sixth herd, 3 years since reintroduction with 190 bison, is added. Find the new line without refitting from scratch.

Solution

The five x values have mean \bar{x} = 3, so the new point sits exactly at the mean of x, and the slope stays 21 bison per year. The old line predicts \hat{y} = 7 + 21(3) = 70 there, which is \bar{y}, so the new herd’s residual is e = 190 - 70 = 120. The intercept rises by \dfrac{120}{6} = 20:

\hat{y} = 27 + 21x

Check: the six sizes add to 350 + 190 = 540, so the new \bar{y} = 90, and 27 + 21(3) = 90, so the line passes through the new center as it must.

The sixth herd is an outlier with no leverage at all. It does not tilt the line, yet it lifts every prediction by 20 bison. Whether that counts as influential depends on the use: a slope is unaffected, a prediction is not.

Boundaries

The 2s guideline and the word “noticeably” are judgments, not tests with a fixed answer. How big a change in slope matters depends on the context and the use of the line.
A point judged from a fit that includes it can hide, as herd C did: it shrinks its own residual and widens s and s_x. Judge a suspect point against the line fitted without it.
Two or more unusual points can shield each other. Removing one at a time may change little while removing the group changes a lot, so refit without the whole group as well.
Removing a point needs a reason outside the numbers: a recording error, or a case from a different situation, like the fenced herd. Without one, report the fit with and without the point. Either way, a line describes only the range of x it was fitted on.
The weighted-average and mean-of-x results are facts about the least-squares line. A line chosen by another rule, such as the smallest sum of absolute residuals, reacts to unusual points differently.

Where this goes next

A far point is sometimes not off the pattern at all but on a curve the line cannot follow; the residual plot shows it, and re-expressing a variable can straighten the curve. A point that changes r also changes r^2, the share of variation the line explains (r squared and the standard deviation of the residuals). Every later use of a fitted line, including inference about the slope, inherits these checks: a slope that one point decides should be reported with that point named.

Exercises

Outliers and leverage

Dawit fits a line to 20 bison calves aged 1 to 11 months, with age x in months and weight y in kilograms: \hat{y} = 30 + 18x, s = 9 kg, \bar{x} = 6 months, s_x = 2.5 months. For each calf in Exercises 1 to 4, find its residual and the z-score of its age, and say whether it is an outlier, whether it has high leverage, and whether adding it would likely change the slope much.

A calf of 5 months weighing 141 kg.
A calf of 7 months weighing 150 kg.
A calf of 16 months weighing 318 kg.
A calf of 15 months weighing 200 kg.
Of the calves in Exercises 1 to 4, which would lift the line but hardly tilt it? Which would change the slope most?

Removing and refitting

Bereket counted the calves born in one season, y, in eight herds of size x (made-up data): (12, 3), (18, 5), (25, 6), (31, 8), (40, 10), (46, 13), (52, 13), (60, 15). Use technology for the fits.

Find the least-squares line, r and s for the eight herds.
A herd of 36 bison with 22 calves is added. Find its residual from the line of Exercise 6, refit, and classify the herd.
Instead, a herd of 150 bison with 10 calves is added. Find the z-score of its size against the eight herds, refit, and compare the two lines’ predictions for a herd of 100 bison. Is the herd influential?
In the nine-herd fit of Exercise 8, s = 3.88 calves. Find the residual of the 150-bison herd from that line and compare it with 2s. What does the comparison show?
Instead, a herd of 150 bison with 37 calves is added. Classify it, and say what happened to r.

A point at the mean of x

A line fitted to seven herds is \hat{y} = 12 + 3x, with \bar{x} = 40. An eighth herd at (40, 160) is added. Find the new line.
A line fitted to nine points is \hat{y} = 50 + 1.5x, with \bar{x} = 20. A tenth point with x = 20 is added, and the intercept becomes 54. Find the tenth point’s y.
Tigist adds an eleventh herd to the ten of Example 1: 53 bison with a range of 100 km². For the ten herds \bar{y} = 162.6. Find the new line.

Reasoning

Tigist says every outlier is influential and every influential point is an outlier. Answer both halves using the lesson’s examples.
Can adding one high-leverage point raise r? Can it lower r? Explain.
Bereket wants to delete herd C from Example 3 because it “ruins the fit”. When is removing a point justified, and what should a report say either way?
A least-squares line \hat{y} = a + bx is fitted to n points with means \bar{x} and \bar{y}. Two points are added, at x = \bar{x} + d and x = \bar{x} - d with d \ne 0, both exactly on the line. Show that the line does not change.
Explain why the 2s guideline, applied to a fit that includes a suspect point, can miss an influential point, and what to do instead.

Answers

Outlier, low leverage, little change in slope. \hat{y} = 30 + 18(5) = 120, residual 141 - 120 = 21 kg, beyond 2s = 18. z = \dfrac{5 - 6}{2.5} = -0.4.
Neither. \hat{y} = 156, residual -6 kg, inside 18; z = 0.4. No real change.
High leverage, not an outlier, little change in slope. \hat{y} = 30 + 18(16) = 318, residual 0; z = \dfrac{16 - 6}{2.5} = 4. It sits on the trend, so it holds the line in place.
High leverage and off the trend, so likely influential. \hat{y} = 300, residual -100 kg; z = 3.6. It would pull the slope down.
The 5-month calf of Exercise 1 lifts the line (big residual, age near \bar{x}); the 15-month calf of Exercise 4 changes the slope most (far out in age and far off the trend).
\hat{y} = 0.10 + 0.254x, r = 0.992, s = 0.59 calves.
Outlier, not influential on the slope. \hat{y} = 0.10 + 0.254(36) = 9.24, residual 22 - 9.24 = 12.76, far beyond 2s = 1.18. Refit: \hat{y} = 1.41 + 0.257x, slope barely moved, though r fell to 0.688.
Influential. z = \dfrac{150 - 35.5}{16.87} \approx 6.79. Refit: \hat{y} = 7.12 + 0.0436x, r = 0.445. For 100 bison: 0.10 + 0.254(100) = 25.5 calves against 7.12 + 0.0436(100) = 11.48 calves, less than half.
\hat{y} = 7.12 + 0.0436(150) = 13.66, residual -3.66, inside 2s = 7.76. The herd pulled the line to itself and widened s; against the eight-herd line its residual is 10 - 38.2 = -28.2, more than 40 times s = 0.59.
High leverage, not influential. z \approx 6.79, residual from the eight-herd line 37 - 38.2 = -1.2. Refit: \hat{y} = 0.41 + 0.245x, a small change in slope, and r rose to 0.999.
\hat{y} = 15.5 + 3x. The old line predicts 132 at x = 40, so e = 28 and the intercept rises by \dfrac{28}{8} = 3.5; the slope stays 3.
y = 120. The rise 4 = \dfrac{e}{10} gives e = 40, and the old line predicts 50 + 1.5(20) = 80 at x = 20.
\hat{y} = 25.04 + 2.488x. The herd sits at \bar{x} = 53, so the slope stays; e = 100 - 162.6 = -62.6, and the intercept drops by \dfrac{62.6}{11} \approx 5.69, from 30.73.
Both halves are false. Herd A had a residual of about 70 km² yet barely tilted the line, so an outlier need not be influential. Herd C changed the slope from 2.488 to 0.989, yet inside the eleven-herd fit its residual was within 2s, so an influential point need not show as an outlier in the fit that includes it.
Yes to both. A far point on the trend stretches the cloud along the line without adding scatter, so r rises (herd B, Exercise 10). A far point off the trend drags the line away from the others and adds large residuals, so r falls (herd C, Exercise 8).
Only with a reason outside the numbers: a recording error, or a case from a different situation, such as a fenced herd. “It ruins the fit” is not a reason, since a line that leaves out awkward cases can mislead. Either way the report names the point, shows the fit with and without it, and states the range of x the chosen line covers.
The new x values add to 2\bar{x}, so \bar{x} is unchanged. On the line, their y values are a + b(\bar{x} \pm d) = \bar{y} \pm bd, which add to 2\bar{y}, so \bar{y} is unchanged. The sum \sum (x_i - \bar{x})(y_i - \bar{y}) gains d(bd) + (-d)(-bd) = 2bd^2 and \sum (x_i - \bar{x})^2 gains 2d^2, so the new slope is \dfrac{bS + 2bd^2}{S + 2d^2} = b, where S is the old bottom sum and bS the old top sum. The intercept \bar{y} - b\bar{x} is unchanged too.
The line is fitted with the point inside it, so a high-leverage point pulls the line toward itself and its residual shrinks; its large residual also widens s, which raises the cutoff. Instead, remove the point, refit, and measure its residual and the change in slope against the line fitted without it.