Precalculus · 8.SP.A.2 · S.ID.B.6A

Judging and comparing models

Two models both run close to the data. How do you tell which one to trust, and how far beyond the data you can trust it?

You need

Where a model holds: domain, interpolation and extrapolationAlgebra 1 Linear, polynomial and exponential growth comparedAlgebra 2

Where you are 5 in Modeling with functions

Functions13

Modeling with functions5

Choosing a function family for data Building a model from a description Interpreting the numbers in a model Where a model holds: domain, interpolation and extrapolation

Judging and comparing models

When you reach for it

You have data and more than one candidate rule for it: a line and an exponential, an exponential and a logistic curve. Differences and ratios (Choosing a function family for data) may already have pointed at a family, and technology (Fitting linear and exponential models with technology) will hand you a best fit inside any family you name. What neither does is tell you how well a finished model matches what was measured, where it starts to miss, or whether to believe it outside the data.

That is this move. Put the models side by side on the same data, compare each prediction with the reading it was meant to match, and read the misses: their size, their signs, and how they change as the input grows.

The procedure

Predicted against actual

For each data point (x, y), a model predicts \hat{y}, its output at that x. The residual is what was measured minus what was predicted:

r = y - \hat{y}

A positive residual means the model came in too low, a negative one too high (Residuals, Interpreting a residual). Laying the readings, each model’s predictions and the residuals in one table puts every model on the same footing: same inputs, same readings, same units.

Reading the pattern

Real readings carry noise: a gauge rounds, a ruler slips. A model that has the right shape leaves residuals that look like that noise: small, with signs mixed and no long runs, and no steady growth in size. A residual plot (residual against input) shows this at a glance. Two patterns mean something is wrong.

A run of one sign, such as an arch or a wave, means the data bend in a way the model cannot. Shifting the model’s numbers will not fix it; the family is wrong.

Residuals growing steadily in one direction mean the model is drifting: it agrees with the data in one stretch and pulls away from it in another. Where the residuals first leave the noise and start to grow is where the model stops describing the situation.

One number: the sum of squared errors

To rank models with a single number, square every residual and add:

\text{SSE} = r_1^2 + r_2^2 + \cdots + r_n^2

Squaring does two jobs. A miss above and a miss below cannot cancel, since both squares are positive, and one large miss counts for more than several small ones. Of two models of the same data, the smaller \text{SSE} is the closer fit, and least squares is exactly the choice of numbers that makes \text{SSE} as small as possible within a family.

The number has limits, and each one matters.

It is in squared output units, and it grows with the number of points. It compares models of the same data in the same units, never two different data sets.
It hides the pattern. The residuals 3, -3, 3, -3 and 0, 0, 0, 6 both give \text{SSE} = 36.
A model with enough adjustable numbers can bring it to 0 (Example 5). A low \text{SSE} then measures how flexible the model is, not how right.
It is computed only on the data. It says nothing about any input outside them.

Inside the data and outside it

A prediction outside the range of the data (Where a model holds: domain, interpolation and extrapolation) can only be judged by what the situation must do there. A draining battery reaches empty at some finite time; a plant stops growing at some height. A model whose long-run behavior matches what the situation must do deserves more trust out there than one with a slightly smaller \text{SSE} inside.

Worked examples

Example 1

Two models for a draining battery

Ganbold streams video on his phone at a fixed brightness and records the battery level each hour (made-up data). With t in hours and B in percent of a full charge, the readings are (0, 100), (1, 91), (2, 83), (3, 72), (4, 64) and (5, 55).

Ganbold proposes the line B = 100 - 9t. Tsetseg proposes B = 100(0.887)^t, an exponential that matches the first and last readings. Compare each model’s predictions with the readings and read the residuals.

Solution

Put each hour into each model, rounding the exponential’s predictions to one decimal place, and subtract each prediction from the reading to get the residual r:

t	B	line	r	exp.	r
0	100	100	0	100	0
1	91	91	0	88.7	2.3
2	83	82	1	78.7	4.3
3	72	73	-1	69.8	2.2
4	64	64	0	61.9	2.1
5	55	55	0	54.9	0.1

The line never misses by more than 1 point, and its two misses have opposite signs. For a gauge that shows whole percents, that is noise.

The exponential’s residuals are all zero or positive and highest in the middle: an arch. Pinned to the data at both ends, the curve sags below every reading in between. The reason is its shape. An exponential with factor 0.887 loses 11.3, then 10.0, 8.9, 7.9 and 7.0 points in successive hours, less each hour, while the phone loses close to 9 every hour.

Figure 1The line’s residuals look like noise. The exponential’s all sit on one side of zero and form an arch, which says the curve has the wrong shape.

Example 2

One number for each battery model

Find the sum of squared errors of each model in Example 1. Then say what happens to both numbers if the battery is recorded as a fraction of a full charge (0.91 rather than 91). Finally, find when each model says the battery is empty.

Solution

Square the residuals in the table and add:

\text{line: } 0 + 0 + 1 + 1 + 0 + 0 = 2

\begin{aligned} \text{exponential: } &0 + 5.29 + 18.49 \\ &+ 4.84 + 4.41 + 0.01 = 33.04 \end{aligned}

Unrounded predictions give 33.3 for the exponential, so call it about 33. The line’s \text{SSE} is about one sixteenth of the exponential’s.

As fractions, every reading and prediction is divided by 100, so every residual is too and every square is divided by 10{,}000. The sums become 0.0002 and about 0.0033. Both numbers changed; the ranking and the ratio did not. An \text{SSE} of 33 is neither large nor small on its own. It means something only beside another model’s \text{SSE} on the same readings.

Outside the data, the line reaches 0 when 100 - 9t = 0, at \displaystyle t = \frac{100}{9} \approx 11.1 hours. The exponential never reaches 0: at 11.1 hours it still predicts about 26\%, and a power of 0.887 is positive for every t. A battery under steady use does run out, so here the model that fits better inside the data is also the one whose long-run behavior is right.

Example 3

Two models for a sorghum plant, early weeks

Tsetseg measures a sorghum plant every week (made-up data). With t in weeks and h in centimeters, the first five readings are (0, 10), (1, 16), (2, 26), (3, 41) and (4, 65). Somto fits the exponential h = 10(1.6)^t. Tsetseg, who knows this variety stops growing near 300 cm, writes the logistic model

h = \frac{300}{1 + 29e^{-0.5t}}

Which model fits these five weeks better?

Solution

t	h	exp.	r	logistic	r
0	10	10	0	10	0
1	16	16	0	16.1	-0.1
2	26	25.6	0.4	25.7	0.3
3	41	41.0	0.0	40.2	0.8
4	65	65.5	-0.5	60.9	4.1

The exponential’s residuals are all within half a centimeter, with mixed signs. Its \text{SSE} is about 0.4.

The logistic does well for three weeks, then falls 4.1 cm short at week 4; its \text{SSE} is about 17.5. The plant is still growing faster than the logistic allows. On these five weeks alone, every test favors the exponential.

Example 4

The later weeks arrive

Tsetseg keeps measuring: 125 cm at week 6, 195 at week 8, 248 at week 10 and 283 at week 12. Find both models’ residuals at these weeks. Where does the exponential start to drift, and which model should be used now?

Solution

t	h	exponential	r
6	125	167.8	-42.8
8	195	429.5	-234.5
10	248	1099.5	-851.5
12	283	2814.7	-2531.7

t	h	logistic	r
6	125	122.8	2.2
8	195	195.9	-0.9
10	248	251.0	-3.0
12	283	279.9	3.1

The exponential’s residuals are all negative and grow several times larger every two weeks. The only hint in Example 3 was week 4, its first negative residual, too small then to mean anything on its own. By week 6 it is 43 cm too high, and by week 12 it predicts a plant over 28 meters tall. The drift starts between weeks 4 and 6 and never turns back, because a fixed growth factor of 1.6 keeps multiplying while the plant slows.

The logistic’s residuals stay within about 3 cm, with mixed signs, the whole way. Over all nine weeks its \text{SSE} is about 42, against about 7.2 million for the exponential.

The ranking flipped. The early \text{SSE} was not wrong; it judged only weeks 0 to 4, where the plant really was growing by a nearly constant factor. The logistic was the better model all along, because its ceiling matched what the plant had to do. Use the logistic.

Figure 2Up to week 4 the exponential fits better. After it, the exponential drifts off and the logistic, with its ceiling, holds.

Example 5

A curve through every point

Somto takes only the first four battery readings of Example 1 (hours 0 to 3) and finds a cubic that passes through all four:

B = -\frac{2}{3}t^3 + \frac{5}{2}t^2 - \frac{65}{6}t + 100

Compare it with the line B = 100 - 9t on those four readings, then at hours 4, 5 and 6.

Solution

The cubic gives 100, 91, 83 and 72 at t = 0, 1, 2, 3. For instance, at t = 3 it gives -18 + 22.5 - 32.5 + 100 = 72. Every residual is 0, so its \text{SSE} is 0. The line’s residuals on the same four hours are 0, 0, 1, -1, so its \text{SSE} is 2. By the number alone, the cubic wins.

Now the hours it was not built on. At t = 4 the cubic predicts 54, against a reading of 64; at t = 5 it predicts 25, against 55. The line predicts both readings exactly. At t = 6 the cubic predicts -19, a battery below empty.

A cubic has four adjustable numbers, and four points can always be matched exactly by choosing them. Its \text{SSE} of 0 measured that freedom, not any truth about the phone: it bent to follow the 11-point drop between hours 2 and 3, where the phone usually lost about 9, a wobble of the kind every set of readings carries, and carried that bend forward into a plunge.

Boundaries

Same data, same units. Residuals and \text{SSE} compare models only on the same data points measured in the same units. A model fitted on a log scale (Semi-log plots: straightening exponential data with logarithms) makes the misses in \log h small, not the misses in h; compare models on the original scale.

More points than adjustable numbers. A polynomial with as many coefficients as there are data points (all at different inputs) passes through every one of them, so it reaches \text{SSE} = 0 whatever the situation. Only a model with fewer numbers than points is tested by the data at all.

Patterns need enough points. With three or four residuals, a run of one sign can happen by chance. An arch or a drift is evidence when it spans many points.

The data end where they end. Small residuals inside the data say nothing beyond the last reading; a drift can begin right there. Outside the data, only knowledge of the situation (a battery empties, a plant stops) can rank models, and it must be real knowledge of the situation, not a preference for a shape.

Where this goes next

Inside one family, Least squares regression chooses the numbers that make \text{SSE} smallest, and in statistics Making and reading residual plots become the standard test of whether a line is the right family. The habit built here, checking a model against readings it was not built on, is how every model is judged, from a phone battery to a population forecast: fit on some data, test on the rest, and trust the model only as far as both the data and the situation reach.

Exercises

Predicted against actual

Somto records his tablet’s battery each hour: 96, 88, 79, 71, 62 at t = 0, 1, 2, 3, 4. Find the residuals and the \text{SSE} for the model B = 96 - 8.5t.
For the same readings, find the residuals (to two decimal places) and the \text{SSE} for B = 96(0.9)^t. Which of the two models fits better?
Somto fits the line h = 25.1t - 16 to all nine sorghum readings of Examples 3 and 4. Find its residuals at weeks 0, 4, 8 and 12, and describe their signs.

Reading residuals

A model’s residuals, in order of input, are 0.3, -0.2, 0.1, 1.5, 4.8, 11.6. Where does the model start to drift, and is it predicting too high or too low?
A model’s residuals, in order of input, are -4, 1, 3, 4, 2, -1, -5. What do they say about the model?

The sum of squared errors

Find the \text{SSE} for the residuals 2, -1, 0, 3, -2.
Find the \text{SSE} of each list of residuals: 2, -2, 2, -2 and 0, 0, 0, 4. Which model would you rather have, and what does that say about the \text{SSE}?
A battery model has \text{SSE} = 0.0004 with the battery recorded as a fraction of a full charge. What is its \text{SSE} with the battery recorded in percent?
A plant model has \text{SSE} = 2500 with heights in centimeters. What is it with heights in meters?
The line of Exercise 3 has \text{SSE} about 1872 over the nine sorghum readings. Compare it with the logistic model of Example 4.

Drift and the range of the data

A second sorghum plant is fitted over weeks 0 to 4 by h = 12(1.5)^t and by \displaystyle h = \frac{240}{1 + 19e^{-0.45t}}. At week 10 the plant measures 195 cm. Find each model’s residual there.
A phone’s battery is modeled by B = 92 - 7.5t and by B = 92(0.92)^t. When does each model say the battery reaches 0\%? When does the exponential say it reaches 5\%?
Somto passes a quadratic through the readings (0, 50), (1, 44) and (2, 40) of a draining battery. Find it, its \text{SSE}, and its prediction at t = 6. When does it say the battery starts to recharge?
Predict the sorghum plant’s height at week 20 with the line of Exercise 3 and with the logistic model of Example 3. Which prediction do you trust, and why?

Reasoning

Explain why the \text{SSE} of a model of one data set cannot be compared with the \text{SSE} of a model of a different data set.
Explain why the residuals are squared before adding, rather than simply added.
Somto says that adding adjustable numbers to a model until its \text{SSE} is 0 always gives the best model. Explain what is wrong, using Example 5.

Answers

Predictions 96, 87.5, 79, 70.5, 62; residuals 0, 0.5, 0, 0.5, 0; \text{SSE} = 0.25 + 0.25 = 0.5.
Predictions 96, 86.4, 77.76, 69.98, 62.99; residuals 0, 1.6, 1.24, 1.02, -0.99; \text{SSE} \approx 6.1. The line fits better: its \text{SSE} is about a twelfth as large, and the exponential’s residuals run positive for three hours, then flip.
Predictions -16, 84.4, 184.8, 285.2, so residuals 26, -19.4, 10.2, -2.2. The signs go positive, negative, positive, negative: a wave, so a line is the wrong family for an S-shaped growth.
It drifts after the third reading, where the residuals leave the noise of a few tenths and grow fast. They are positive, so the readings are above the model: it predicts too low.
An arch: negative at both ends and positive in the middle. The data bend in a way the model cannot, so the model belongs to the wrong family, even though no single miss is large.
4 + 1 + 0 + 9 + 4 = 18.
Both are 4 + 4 + 4 + 4 = 16 and 0 + 0 + 0 + 16 = 16. The first list alternates like noise; the second is perfect for three points and then misses badly, which may be the start of a drift. Equal \text{SSE} can hide very different behavior, so read the residuals too.
Each residual is 100 times larger, each square 10{,}000 times: \text{SSE} = 4.
Each residual is divided by 100, each square by 10{,}000: \text{SSE} = 0.25.
The logistic’s \text{SSE} is about 42 on the same nine readings, so the line’s is about 45 times as large. With the wave of Exercise 3, both the number and the pattern favor the logistic.
Exponential: 12(1.5)^{10} \approx 692.0, residual 195 - 692.0 = -497.0. Logistic: \displaystyle \frac{240}{1 + 19e^{-4.5}} \approx 198.2, residual -3.2.
The line: 92 - 7.5t = 0 at \displaystyle t = \frac{92}{7.5} \approx 12.3 hours. The exponential never reaches 0, since (0.92)^t > 0. It reaches 5\% when \displaystyle (0.92)^t = \frac{5}{92}, so \displaystyle t = \frac{\ln(5/92)}{\ln 0.92} \approx 34.9 hours.
The drops are 6 and 4, so the second difference is 2 and the t^2 coefficient is 1: B = t^2 - 7t + 50. It passes through all three points, so \text{SSE} = 0. At t = 6 it gives 36 - 42 + 50 = 44. Its vertex is at t = 3.5, so after 3.5 hours it says the battery recharges itself.
Line: 25.1(20) - 16 = 486 cm. Logistic: \displaystyle \frac{300}{1 + 29e^{-10}} \approx 299.6 cm. Trust the logistic: the plant has a ceiling near 300 cm, the line has none, and the line’s residuals already showed a wave.
\text{SSE} is in squared units of the output and grows with every point added, so its size depends on the units and the number of points, not only on how good the fit is. Only models of the same readings in the same units share a scale.
Simply added, positive and negative residuals cancel: residuals 5 and -5 add to 0 though the model missed twice. Squares are never negative, so every miss counts, and large misses count more.
With as many adjustable numbers as points, the model can pass through every point, so \text{SSE} = 0 shows only that it was flexible enough. Example 5’s cubic had \text{SSE} = 0 on four readings, then missed the next two by 10 and 30 points and predicted a negative battery. The test of a model is data it was not built on.