Statistics

Bias and variability of an estimator

Several formulas turn the same sample into an estimate. Which one aims at the truth, and which one lands close to it?

You need

Sampling variability and simulating itMath 7

Unlocks

Standard errorStatistics

Where you are 7 in Sampling distributions

Statistics7

Distributions8

Sampling distributions7

Bias and variability of an estimator

Sampling variability and simulating it Simulating a sampling distribution and a margin of error The ten percent condition Sampling distribution of a sample mean ▾ Show all 7 (2 more)

Why this exists

A city’s rental scooters carry serial numbers 1, 2, 3, \dots, N, and the company will not say how large N is. Sophea writes down the numbers on five scooters picked at random:

61, \quad 187, \quad 233, \quad 302, \quad 415

(made-up data). Each of three people turns the same five numbers into a different estimate of N. Vannak says 415: the largest number seen. Bopha doubles the sample mean. Sophea adds to the largest number an allowance for the scooters above it. All three are reasonable, and they disagree.

One sample cannot settle it, because nobody knows N. What can be judged is the method. Sampling variability showed that a statistic, recomputed over many samples, makes a pile with a center and a spread. Lay that pile next to the parameter: does it sit on the parameter or to one side, and how widely does it scatter? Those two questions, bias and variability, are how estimators are compared.

Definition

An estimator is a rule that turns a sample into a guess at a parameter, such as “take the largest number”. The value it gives for one sample is an estimate, written with a hat: \hat{N} = 415.

The estimator’s sampling distribution is the pile of estimates from every possible random sample of the same size from the same population, each sample counted once (all samples equally likely).

The bias of an estimator is the mean of its sampling distribution minus the parameter. An estimator is unbiased when its bias is 0: its sampling distribution is centered at the parameter. A negative bias means it runs low on average, a positive bias high.
The variability of an estimator is the spread of its sampling distribution, measured by that distribution’s standard deviation.

Bias and variability are properties of the rule and the sampling method together, never of one estimate. A single unbiased estimate can be far off; it is the rule that aims true.

Intuition

The scooter question has a small version that can be worked out completely. Take a fleet of only N = 6 scooters and samples of 2. There are just 15 possible samples, and Figure 1 shows, for every one of them, three estimates:

the sample maximum, \hat{N} = \max;
twice the sample mean, less 1, \hat{N} = 2\bar{x} - 1 (the -1 is explained under Results);
the adjusted maximum, \displaystyle \hat{N} = \frac{n + 1}{n}\max - 1, which for n = 2 is \displaystyle \frac{3}{2}\max - 1.

Figure 1Every possible sample of 2 scooters from a fleet of 6, and the three estimates each sample gives. The triangle marks the true size, 6. Only the maximum’s pile sits to one side of it.

The maximum’s pile stops at 6 and leans left of it: no sample can show a number larger than N, and most samples miss scooter 6. That is bias. The other two piles balance on the triangle at 6: unbiased. Of those two, the adjusted maximum’s dots are packed more tightly, so it has less variability.

Picture an archer. Bias is aiming to one side of the bull’s-eye; variability is how widely the arrows scatter around wherever they aim. An estimator can have either fault, both, or neither, and a little bias with far less scatter can beat a true aim with wild scatter.

A larger sample tightens every pile (Figure 2), because each estimate rests on more information. It does not move the aim of a collection method that leans to one side.

Figure 2Twice the sample mean, less 1, for every sample of 2 and every sample of 4 from the same fleet of 6. Both piles center on 6; the larger samples crowd closer to it.

Results

For a random sample of n different serial numbers from 1, \dots, N:

\text{mean of } \bar{x} = \frac{N + 1}{2}

\text{mean of } \max = \frac{n(N + 1)}{n + 1}

So 2\bar{x} - 1 is unbiased for N, the maximum is biased low when n < N, and \displaystyle \frac{n + 1}{n}\max - 1 is unbiased. Written as \displaystyle \max + \frac{\max - n}{n}, the adjustment adds one average gap: the \max - n unseen numbers below the maximum fall into n gaps, and one more gap of the same size is assumed above it.

For n independent draws from a population with variance \sigma^2, the sample variance from Computing a standard deviation, sample and population,

s^2 = \frac{\sum (x - \bar{x})^2}{n - 1},

is unbiased for \sigma^2. Dividing the same sum by n gives an estimator whose mean is \displaystyle \frac{n - 1}{n}\sigma^2, biased low. The sample standard deviation s is still slightly biased low for \sigma.

A larger random sample shrinks an estimator’s variability. It does not remove bias built into how the sample is collected.

Why it is true

Twice the mean, less 1. Each scooter in the sample is equally likely to be any of the N, so each sample value averages \displaystyle \frac{1 + 2 + \dots + N}{N} = \frac{N + 1}{2}. The mean of \bar{x} is the average of those means, \displaystyle \frac{N + 1}{2}, and doubling and subtracting 1 gives a mean of N. Without the -1, 2\bar{x} would average N + 1.

The maximum. A sample of n has maximum m when it holds m and n - 1 numbers below m, which can happen in \binom{m - 1}{n - 1} ways out of \binom{N}{n}. Since m\binom{m - 1}{n - 1} = n\binom{m}{n}, and the column sum \binom{n}{n} + \binom{n + 1}{n} + \dots + \binom{N}{n} equals \binom{N + 1}{n + 1},

\begin{aligned} \text{mean of } \max &= \frac{n\binom{N + 1}{n + 1}}{\binom{N}{n}} \\ &= \frac{n(N + 1)}{n + 1}. \end{aligned}

Why n - 1 is exactly right. Write each deviation from \mu as (x - \bar{x}) + (\bar{x} - \mu) and square. The cross terms add to 2(\bar{x} - \mu)\sum(x - \bar{x}) = 0, because deviations from \bar{x} add to 0. So

\sum (x - \mu)^2 = \sum (x - \bar{x})^2 + n(\bar{x} - \mu)^2.

Average both sides over all samples. Each (x - \mu)^2 averages \sigma^2, so the left side averages n\sigma^2. For independent draws, \bar{x} has variance \displaystyle \frac{\sigma^2}{n} (taken up in Sampling distribution of a sample mean), so n(\bar{x} - \mu)^2 averages \sigma^2. That leaves

\text{mean of } \sum (x - \bar{x})^2 = (n - 1)\sigma^2.

The sum of squared deviations from \bar{x} is short by exactly one \sigma^2, the share used up by measuring from \bar{x} instead of \mu. Dividing by n - 1 gives a mean of \sigma^2 exactly.

Why s is still biased. For any estimator, (\text{mean})^2 = \text{mean of the square} - \text{variance}. With s, the mean of s^2 is \sigma^2, and s varies, so (\text{mean of } s)^2 < \sigma^2 and the mean of s is below \sigma. Taking a square root does not carry unbiasedness along.

Worked examples

Example 1

Three estimates from one sample

Find the three estimates of N from Sophea’s sample 61, 187, 233, 302, 415.

Solution

The sample maximum is \hat{N} = 415.

For the mean, the numbers add to 61 + 187 + 233 + 302 + 415 = 1198, so \displaystyle \bar{x} = \frac{1198}{5} = 239.6 and \hat{N} = 2(239.6) - 1 = 478.2.

For the adjusted maximum, n = 5: \displaystyle \hat{N} = \frac{6}{5}(415) - 1 = 498 - 1 = 497. As a gap: 415 - 5 = 410 unseen numbers lie below 415 in 5 gaps of 82 on average, and 415 + 82 = 497.

The estimates are 415, 478.2 and 497 scooters. This one sample cannot say which is best; that takes the sampling distributions, next.

Example 2

Every sample from a fleet of six

For a fleet of N = 6 and samples of 2, list every sample with its three estimates and find the mean of each estimator.

Solution

Each sample is a pair of different numbers; there are \binom{6}{2} = 15, all equally likely.

Sample	\max	2\bar{x} - 1	\displaystyle \frac{3}{2}\max - 1
1, 2	2	2	2
1, 3 / 2, 3	3	3 / 4	3.5
1, 4 / 2, 4 / 3, 4	4	4 / 5 / 6	5
1, 5 to 4, 5	5	5 / 6 / 7 / 8	6.5
1, 6 to 5, 6	6	6 / 7 / 8 / 9 / 10	8

For a pair a, b, 2\bar{x} - 1 = a + b - 1.

The maxima add to 2 + 2(3) + 3(4) + 4(5) + 5(6) = 70, so the mean of the maximum is \displaystyle \frac{70}{15} \approx 4.67. Its bias is 4.67 - 6 = -1.33 scooters, matching \displaystyle \frac{n(N + 1)}{n + 1} = \frac{2 \cdot 7}{3}.

The values of 2\bar{x} - 1 add to 90, and so do the adjusted maxima: 2 + 2(3.5) + 3(5) + 4(6.5) + 5(8) = 90. Both means are \displaystyle \frac{90}{15} = 6, the true N. These two are unbiased; the maximum runs low by 1.33 scooters on average.

Example 3

Comparing spread

For the same 15 samples, find the standard deviation of each estimator. Which would you use?

Solution

The 15 samples are the whole sampling distribution, not a sample from it, so each variance divides by 15.

For 2\bar{x} - 1, the squared distances from 6 are 16, 9, 4, 4, 1, 1, 0, 0, 0, 1, 1, 4, 4, 9, 16, adding to 70: variance \displaystyle \frac{70}{15} \approx 4.667, standard deviation 2.160.

For the adjusted maximum, the distances from 6 are -4, -2.5, -1, 0.5, 2 with counts 1, 2, 3, 4, 5: 16 + 2(6.25) + 3(1) + 4(0.25) + 5(4) = 52.5, variance 3.5, standard deviation 1.871.

For the maximum, the distances from its own mean \displaystyle \frac{14}{3} give variance \displaystyle \frac{14}{9}, standard deviation 1.247.

The maximum has the least spread but sits off center. Between the two unbiased estimators, the adjusted maximum scatters less (1.871 against 2.160 scooters), so it is the better choice: aimed true, with tighter shots. It uses the most informative number in the sample, while the mean blends in small numbers that say little about where the fleet ends.

Example 4

A larger sample

From the same fleet, find the mean and standard deviation of 2\bar{x} - 1 for samples of 4, and compare with samples of 2.

Solution

A sample of 4 leaves out a pair, so its sum is 21 - (\text{sum of the left-out pair}). The left-out pair runs over all 15 pairs, with sums 3 to 11 centered at 7. So the sum of four averages 21 - 7 = 14, and \displaystyle 2\bar{x} - 1 = \frac{14}{2} - 1 = 6 on average: still unbiased.

Each estimate is \displaystyle \frac{\text{sum}}{2} - 1, so it varies half as widely as the pair sums, whose variance is \displaystyle \frac{70}{15}. The variance is \displaystyle \frac{1}{4} \cdot \frac{70}{15} = \frac{7}{6}, standard deviation 1.080.

Doubling the sample from 2 to 4 cut the standard deviation from 2.160 to 1.080 scooters, with the center fixed at 6 (Figure 2).

Example 5

Bias that size does not cure

Bopha samples only at the downtown hub, where the fleet of six keeps scooters 4, 5 and 6. Find the mean of 2\bar{x} - 1 for her samples of 2, then for samples of 3.

Solution

Her possible samples of 2 are 4, 5 and 4, 6 and 5, 6, giving 8, 9 and 10, with mean 9: a bias of +3 scooters.

With 3 scooters she always gets all three, so \bar{x} = 5 and \hat{N} = 9 every time. The variability is now 0, and the bias is still +3. The larger sample sharpened the estimate around the wrong value, because the hub is not a random sample of the fleet (see Bias). The formula is unbiased only under the sampling method it was built for.

Example 6

Why the sample variance divides by n − 1

Three scooters made 2, 4 and 6 trips today. A rider picks one at random, then a second rider picks at random from all three again, so the same scooter can come up twice. List all samples of 2 and find the mean of s^2, of the divide-by-n version, and of s.

Solution

The population has \mu = 4 and \displaystyle \sigma^2 = \frac{4 + 0 + 4}{3} = \frac{8}{3}, so \sigma \approx 1.633.

There are 3 \times 3 = 9 ordered samples. For two values, \bar{x} is halfway, each deviation is \displaystyle \frac{a - b}{2}, and the sum of squared deviations is \displaystyle \frac{(a - b)^2}{2}. So \displaystyle s^2 = \frac{(a - b)^2}{2} and the divide-by-n version is \displaystyle \frac{(a - b)^2}{4}.

Gap a - b	Samples	s^2	Divide by n	s
0	3	0	0	0
\pm 2	4	2	1	\displaystyle \sqrt{2}
\pm 4	2	8	4	\displaystyle 2\sqrt{2}

Mean of s^2: \displaystyle \frac{4(2) + 2(8)}{9} = \frac{24}{9} = \frac{8}{3}, exactly \sigma^2.

Mean of the divide-by-n version: \displaystyle \frac{4 + 8}{9} = \frac{4}{3}, half of \sigma^2, matching \displaystyle \frac{n - 1}{n} = \frac{1}{2}.

Mean of s: \displaystyle \frac{4\sqrt{2} + 4\sqrt{2}}{9} = \frac{8\sqrt{2}}{9} \approx 1.257, below \sigma \approx 1.633. The square root of an unbiased estimator is not unbiased.

Boundaries

Bias belongs to the rule and the sampling method together. The formula 2\bar{x} - 1 is unbiased only for a random sample of scooters numbered 1 to N with no gaps; a hub that holds certain scooters, or serial numbers that skip, break it.
Unbiased does not mean close. An unbiased estimate can be impossible: 2\bar{x} - 1 can come out below the largest number already seen.
Unbiasedness does not survive a nonlinear change: s^2 is unbiased for \sigma^2, yet s runs low for \sigma.
s^2 is unbiased for independent draws. Without replacement from a small population it runs high; from the trips 2, 4, 6 without replacement, its mean is 4, not \displaystyle \frac{8}{3}. When the sample is a small part of the population the difference is negligible (The ten percent condition).
A larger sample cuts variability for these estimators, but it never fixes bias from how the sample was collected. The maximum’s own bias, \displaystyle \frac{n - N}{n + 1}, shrinks as n grows, yet is never 0 while n < N.

Where this goes next

The spread of an estimator’s sampling distribution is what Standard error estimates from a single sample. The three sampling-distribution lessons, for a proportion, a mean and a difference of means, show that each of those statistics is unbiased and give a formula for its spread, so the question “aimed true, and how tight?” gets an exact answer there. The overview is at Sampling distributions.

Exercises

Estimates from one sample

Vannak records scooters 38, 112, 205, 260. Find the sample maximum, twice the sample mean less 1, and the adjusted maximum.
Bopha records scooters 12, 30, 350. Find twice the sample mean less 1, and say why it is a poor estimate here.

Every sample from a small fleet

A fleet has N = 5 scooters. List all samples of 2 and find the mean and bias of the sample maximum.
For the same fleet and samples, find the mean of 2\bar{x} - 1.
For the same fleet and samples, find the standard deviations of 2\bar{x} - 1 and of the adjusted maximum \displaystyle \frac{3}{2}\max - 1. Which unbiased estimator is better?
A fleet has N = 4 scooters. Over all samples of 2, find the mean of 2\bar{x} (with no -1) and its bias.

Sample size and bias

For a fleet of 6 and samples of 4, count the samples whose maximum is 4, 5 and 6. Find the mean and bias of the maximum, and compare with samples of 2.
For the same samples of 4, find the mean and standard deviation of the adjusted maximum \displaystyle \frac{5}{4}\max - 1. Compare with its standard deviation for samples of 2.
Vannak samples only at the riverside lot, which holds scooters 1, 2 and 3 of a fleet of 6. Find the mean of the adjusted maximum for his samples of 2 and for his samples of 3.

Dividing by n or n − 1

Two scooters made 1 and 5 trips. Draw 2 with replacement. List the samples and find the mean of s^2, of the divide-by-n version, and of s. Compare with \sigma^2 and \sigma.
Three scooters made 0, 1 and 5 trips. Find \sigma^2, and show that s^2 averages \sigma^2 over all 9 samples of 2 drawn with replacement.
A sample 3, 7, 8 comes from a population with \mu = 5. Find \sum (x - \mu)^2, \sum (x - \bar{x})^2 and n(\bar{x} - \mu)^2, and check the identity in Why it is true.
For samples of 5 independent draws, what fraction of \sigma^2 does the divide-by-n estimator average? What is its mean when \sigma^2 = 20?

Reasoning

Explain why the sample maximum is biased for N whenever n < N, without any formula.
Exercise 2 gave an unbiased estimator a value that cannot be right. Explain how both can be true.
Sophea plans to fix a survey of scooters taken only at the hub by recording 50 scooters instead of 5. Explain what this will and will not change.
In your own words, explain why s is biased low for \sigma even though s^2 is unbiased for \sigma^2.
Estimator A is unbiased with standard deviation 40. Estimator B has bias 5 and standard deviation 10. Which would you use, and why?

Answers

Maximum 260. Sum 615, \bar{x} = 153.75, 2\bar{x} - 1 = 306.5. Adjusted, n = 4: \displaystyle \frac{5}{4}(260) - 1 = 324.
Sum 392, \bar{x} \approx 130.67, \displaystyle 2\bar{x} - 1 = \frac{781}{3} \approx 260.3. Scooter 350 exists, so N \ge 350; the estimate is impossible. The two small numbers dragged the mean down.
Ten samples; maximum 2 once, 3 twice, 4 three times, 5 four times. Mean \displaystyle \frac{2 + 6 + 12 + 20}{10} = 4, bias 4 - 5 = -1 scooter.
The values a + b - 1 are 2, 3, 4, 5, 4, 5, 6, 6, 7, 8, adding to 50; mean 5 = N.
2\bar{x} - 1: squared distances from 5 add to 30, variance 3, standard deviation \displaystyle \sqrt{3} \approx 1.732. Adjusted maximum: values 2, 3.5, 5, 6.5 with counts 1, 2, 3, 4, mean 5, squared distances add to 22.5, variance 2.25, standard deviation 1.5. Both unbiased; the adjusted maximum spreads less, so it is better.
Pair sums 3, 4, 5, 5, 6, 7; 2\bar{x} equals the sum, mean \displaystyle \frac{30}{6} = 5. Bias +1, which is why the -1 is there.
Maximum 4: 1 sample; 5: \binom{4}{3} = 4; 6: \binom{5}{3} = 10. Mean \displaystyle \frac{4 + 20 + 60}{15} = 5.6, bias -0.4, against -1.33 for samples of 2. Smaller, but not 0.
Values 4, 5.25, 6.5 with counts 1, 4, 10; mean \displaystyle \frac{90}{15} = 6, unbiased. Variance \displaystyle \frac{4 + 4(0.5625) + 10(0.25)}{15} = \frac{7}{12}, standard deviation 0.764, against 1.871 for samples of 2.
Samples 1, 2 and 1, 3 and 2, 3 give 2, 3.5, 3.5: mean 3, bias -3. A sample of 3 always has maximum 3, so \displaystyle \frac{4}{3}(3) - 1 = 3: bias still -3, now with no spread.
Samples 1, 1 / 1, 5 / 5, 1 / 5, 5 give s^2 = 0, 8, 8, 0, mean 4; \sigma^2 = 4. Divide by n: mean 2. \displaystyle s = 0, 2\sqrt{2}, 2\sqrt{2}, 0, mean \displaystyle \sqrt{2} \approx 1.414, below \sigma = 2.
\mu = 2, \displaystyle \sigma^2 = \frac{4 + 1 + 9}{3} = \frac{14}{3}. Squared gaps for the unequal pairs are 1, 25, 16, each twice: 84. Mean of \displaystyle s^2 = \frac{(a - b)^2}{2} is \displaystyle \frac{84}{18} = \frac{14}{3}.
\sum (x - \mu)^2 = 4 + 4 + 9 = 17. \bar{x} = 6, \sum (x - \bar{x})^2 = 9 + 1 + 4 = 14. n(\bar{x} - \mu)^2 = 3. And 14 + 3 = 17.
\displaystyle \frac{n - 1}{n} = \frac{4}{5}; mean \displaystyle \frac{4}{5}(20) = 16.
No sample can contain a number above N, so the maximum is never too high, and any sample missing scooter N gives a maximum too low. Its average is pulled below N.
Unbiased means the estimates average N over all samples. Some samples land high and some land low, and a sample with two very small numbers lands low enough to fall under its own maximum.
A larger sample makes the estimates vary less, but every one still comes from the hub, so the estimates crowd around the wrong value. Only a random sample of the whole fleet removes the bias.
Values of s vary, so the square of their average is less than the average of their squares. The average of s^2 is \sigma^2, so the average of s is less than \sigma.
B: its typical miss is about 5 plus a scatter of 10, while A’s estimates scatter about 40 either side of the truth. Small bias is worth a fourfold cut in spread here.