Computing a standard deviation, sample and population

When you reach for it

You have a list of numbers and want one number for how far they typically sit from their mean, in the data’s own unit: pages, years, hours. The variance is the mean of the squared deviations, but it is in squared units. Its square root, the standard deviation (Standard deviation), is back in the data’s unit, and it is the measure of spread used throughout statistics.

There are two versions. When the numbers are the whole group you care about, a population, divide the sum of the squared deviations by the count, as the variance lesson did. When the numbers are a sample standing in for a bigger group, divide by one less than the count. This lesson runs over two days: Day 1 is the population standard deviation, Day 2 the sample one and how to choose between them.

The procedure

Day 1: the population standard deviation

For a whole population of N values, the mean is written \mu (mu) and the standard deviation \sigma (sigma). Greek letters mark numbers that describe a whole population, as Parameters and statistics explains. The standard deviation is the square root of the variance:

Read it from the inside out. x - \mu is one value’s deviation from the mean. Squaring removes its sign. \sum, a capital sigma, means “add this up over every value”. Dividing by N gives the variance \sigma^2, and the square root brings the unit back.

A table with three columns, value, deviation and squared deviation, keeps the work in order, and its middle column carries a free check: the deviations always add to 0. Keep the mean exact and round only the final answer. A rounded mean makes every deviation slightly wrong, and Example 2 shows the answer drifting.

When values repeat, a frequency table lists each different value once with its count f. A value that appears f times contributes its squared deviation f times, so multiply each squared deviation by its count before adding. Then divide by N, the total of the counts, not by the number of rows.

Two facts check that an answer is sensible:

• \sigma is never larger than the largest distance from the mean. Call that distance D. Every squared deviation is at most D^2, so their mean is at most D^2 and its square root is at most D. It equals D only when every value is exactly D from the mean.
• \sigma is never negative, and \sigma = 0 exactly when all the values are equal. A sum of squares is 0 only when every square is 0, that is, when every value equals the mean.

Day 2: the sample standard deviation

Often the group of interest is too big to measure, or measuring uses it up: finding a torch’s battery life means running it flat. Then a sample is measured instead. Its mean is written \bar{x}, its standard deviation s, and its size n. The table is the same; only the divisor changes:

Why one less. The aim is the spread of the whole group, measured from the group’s mean \mu, which nobody knows. The deviations are taken from \bar{x} instead, and \bar{x} is worked out from the sample itself, so it sits right in the middle of the sample: no other number has a smaller sum of squared distances to those values. The true mean \mu is almost always a little off from \bar{x}, and squared distances measured from it would add to more. So deviations from the sample’s own mean come out a little small on average, and dividing by n - 1 instead of n makes the result a little bigger to make up for it. Example 5 shows this in numbers. Why n - 1 is exactly the right amount is taken up later, in Bias and variability of an estimator.

For the same numbers, then, s is a little larger than the figure from dividing by n. Dividing the same sum by n - 1 instead of n multiplies the variance by \displaystyle \frac{n}{n - 1}, so s is \displaystyle \sqrt{\frac{n}{n - 1}} times the divide-by-n figure. For n = 2 that factor is \displaystyle \sqrt{2} \approx 1.41, for n = 10 about 1.05, and for n = 100 about 1.005. The gap shrinks as the sample grows.

Which one to use depends on the question, not on the numbers. If the values are every member of the group you want to describe, they are a population: use \sigma and divide by N. If they stand in for a bigger group you want to learn about, they are a sample: use s and divide by n - 1. The same five numbers can be either (Example 6).

Calculators and spreadsheets have both. A calculator in statistics mode usually shows a population standard deviation (often labeled \sigma_x) and a sample one (often s_x). A spreadsheet has STDEV.P for a population and STDEV.S for a sample. They are the two formulas above, so a hand computation and the machine agree once the right one is picked.

Worked examples

Boundaries

• A standard deviation needs quantitative data, numbers whose differences mean something.
• The sample formula needs at least two values. With n = 1 the divisor n - 1 is 0, and a single value shows no spread to measure.
• Dividing by n - 1 corrects for measuring from \bar{x}; it cannot fix a sample chosen badly. The sample has to be a fair, random choice from the bigger group.
• Data grouped into bins, as for a histogram, have lost their exact values. Using each bin’s middle in a frequency table then gives only an approximate standard deviation.
• Values far from the rest pull the standard deviation up, as the two outer books did in Example 1. When a Box plots that show outliers flags outliers, read the standard deviation with care.
• Round only the final answer; a rounded mean drifts the result (Example 2).

Where this goes next

Interpreting a standard deviation reads the number in the setting’s own units and says what it does and does not tell you. What adding a constant to every value, or multiplying every value, does to \sigma is the subject of Shifting and rescaling a data set. The Mean absolute deviation stays a simpler measure of spread. Where a single value stands can be said by rank, as a percentile, or by how many standard deviations it sits from the mean, which is where the standard deviation goes next.

Exercises

Day 1: population standard deviation

All data are made up. Treat each set as a whole population, and give \sigma to two decimal places unless it is exact.

1. 3, 4, 8, 10, 15
2. The page counts of every book in a five-book series: 210, 250, 270, 290, 330.
3. 2, 3, 7, 8, 9, 10
4. The bedrooms in each of the 20 flats in a building: 4 flats have 1, 8 have 2, 6 have 3 and 2 have 4.
5. The goals scored in a season by each of the 10 players in a team: 2 players scored 0, 3 scored 1, 3 scored 2 and 2 scored 3.
6. For the population 1, 2, 4, 6, find \sigma with the exact mean. Then redo it with the mean rounded to the nearest whole number, and compare.
7. Could the population 10, 12, 20 have \sigma = 18.7? Explain without computing \sigma. Then find \sigma, and say which number in the working is about 18.7.
8. Find \sigma for each population. A: 5, 5, 5, 5. B: 4, 5, 5, 6. C: 2, 5, 5, 8.

Day 2: sample standard deviation

Treat each set as a sample, and give s to two decimal places unless it is exact.

9. 4, 6, 7, 9, 14
10. The masses of 5 apples picked from an orchard’s harvest: 148, 150, 153, 155, 159 grams.
11. For the numbers in Exercise 9, find the divide-by-n figure too, and the ratio of s to it.
12. The masses of 4 parcels taken from the 300 sent one day: 1.2, 1.5, 1.6, 2.1 kg.
13. A sample of 12 households in a town is asked how many cars each owns: 2 have 0, 4 have 1, 4 have 2 and 2 have 3.
14. Say whether \sigma (divide by N) or s (divide by n - 1) fits, and why. (a) The heights of every player in a basketball team, to describe that team. (b) The ages of 40 people chosen at random in a city, to learn about the city. (c) 8 bolts measured from a run of 10{,}000. (d) The rainfall on every day of last March, to describe last March.
15. A spreadsheet holding 10 values reports a population standard deviation of 4.0. What will its sample standard deviation be?
16. Find the factor \displaystyle \sqrt{\frac{n}{n - 1}} by which s exceeds the divide-by-n figure, to three decimal places, for n = 2, n = 5 and n = 100.

Reasoning

17. Explain why a population’s \sigma is never larger than the largest distance of a value from the mean. When is it equal to it?
18. Explain why, for the same numbers (not all equal), s is always larger than the divide-by-n figure, and why the gap shrinks as n grows.
19. Alessio tests a sample of one torch, and his calculator shows no sample standard deviation. Explain why the formula breaks, and why that makes sense.