Sample size versus population
A Socratic walk-through of sample size versus population — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does a poll of a thousand people describe a nation of millions no less accurately than it describes a small town?
Almost everyone's first instinct about polling is the same: a thousand people is a fifth of a town of five thousand, and a three-thousandth of a percent of a country of three hundred million, so surely the poll tells you far more about the town. It feels like arithmetic. It is wrong, and the reason it is wrong is the whole substance of sampling theory. Where does the intuition go astray?
Reasoning it through
REASONING #Let us build the estimate from scratch and watch what the population size actually does.
Suppose some fraction of people would answer yes to a question, and call that unknown fraction p. You draw n people at random and count the yes answers. Each draw is essentially a coin with unknown bias p. Your estimate is the proportion of heads.
Now ask the only question that matters: how much would that proportion bounce around if you drew a fresh sample? That spread is the standard error, and for a proportion it is the square root of p(1 − p)/n.
Look hard at that expression. There is a p in it and there is an n in it. There is no N — nothing for the size of the population you drew from. The precision of the estimate is governed by how many coins you flipped, not by how many coins were in the bucket.
Put numbers on it, taking the worst case p = 0.5, where the spread is largest. With n = 1,000 the standard error is 0.5/√1000 ≈ 0.0158, about 1.6 percentage points. The conventional 95% margin of error multiplies that by 1.96, giving roughly ±3.1 points. That figure is where the familiar "plus or minus three points" comes from — and notice that computing it never required knowing the country's population.
Does that mean the population is irrelevant? Not quite — there is a correction, and its shape is instructive. When you sample without replacement, the true standard error is multiplied by the finite population correction, √((N − n)/(N − 1)). Feed it a country: with N = 5,000,000 and n = 1,000, the factor is about 0.99990. Feed it the United States and it is closer still to 1. The correction exists; it does nothing.
When does it do something? Only when your sample is a substantial fraction of the population. Sample 1,000 from a town of 5,000 and the factor is √(4000/4999) ≈ 0.894 — an eleven percent improvement, so ±3.1 becomes about ±2.8. Sample 1,000 from a village of 2,000 and the factor is about 0.707. Sample the whole population and it becomes zero, which is correct: a census has no sampling error at all.
So the intuition was not baseless — it was just backwards about magnitude. The fraction sampled matters, but only once it is a large fraction, and 1,000 out of 5,000,000 is not one. And this is why the small town gets the better poll from the same thousand interviews, not the country — which is not what anyone's intuition predicts either.
One more consequence worth having. Since the error falls as √n, halving it costs four times the sample. Going from ±3.1 to ±1.55 points means moving from 1,000 respondents to 4,000. That is why polls cluster around a thousand: it is roughly where the curve stops rewarding you.
The analogy
THE ANALOGY #Think of tasting a pot of soup. You stir it and take a spoonful. To judge the seasoning you need a spoon big enough to be representative — one drop tells you little. But once the spoon is a decent size, it makes no difference whether the pot holds two litres or two hundred; you do not need a bigger spoon for the bigger pot. What you need, absolutely and above everything, is that the pot was stirred.
Soup can be stirred to genuine uniformity, whereas a population cannot — so the whole burden falls on the randomisation of who is contacted and who answers, and unlike stirring, failure there is invisible in the numbers and is not reduced by taking a larger spoonful.
Clarifying the model
THE MODEL #Three refinements hold this together.
First, the margin of error describes sampling variation only. It assumes the sample is a genuine random draw from the population you care about. Everything that violates that — people who do not answer the phone, who are unreachable, who refuse, who are on a frame that omits some group — produces error that the ±3.1 does not include and that more interviews do not shrink. When published polls miss badly, it is nearly always this, not sampling variation.
Second, the ±3 points applies to a single overall proportion at p near 0.5. Sub-groups are smaller samples: the 150 young rural voters inside your 1,000 carry a margin near ±8, and reporting their movement as news is a standard error of practice.
Third, real surveys rarely use simple random sampling. Clustering and weighting change the effective precision — usually worsening it — by a factor called the design effect, so the honest margin is often somewhat wider than the textbook formula gives.
A picture of it
THE PICTURE #How to readThe horizontal axis is how many people you actually interviewed; the vertical axis is the resulting 95% margin of error in percentage points, at the worst-case split of fifty-fifty. Trace the curve and notice two things. It drops steeply and then flattens — the thousandth respondent buys far less than the hundredth, and quadrupling the sample only halves the margin. And notice what is not on this chart: the population being sampled from never appears, because the curve is identical for a town, a country, or a continent.
What became clearer
WHAT CLEARED #Precision comes from the absolute number of independent observations, not from the share of the population they represent. The population size enters only through a correction that is negligible until the sample becomes a serious fraction of the whole — and when it does, it makes the estimate better, not worse. The real fragility of a poll was never its size relative to the nation. It is whether the thousand people were a fair draw, and that is a question the margin of error is silent about.
Where to go next
ONWARD #- How weighting and design effects change the effective sample size in practice.
- Why non-response bias has grown as response rates collapsed, and what pollsters do about it.
Key terms
TERMS #| Term | What it means |
|---|---|
| Standard error | the typical spread of an estimate across repeated samples; for a proportion, √(p(1 − p)/n). |
| Margin of error | a stated interval around an estimate, conventionally 1.96 standard errors for 95% confidence. |
| Finite population correction | the factor √((N − n)/(N − 1)) reducing the standard error when the sample is a large fraction of the population. |
| Design effect | the ratio by which clustering, stratification, and weighting inflate or deflate variance relative to simple random sampling. |
| Non-response bias | error arising when those who answer differ systematically from those who do not; unaffected by sample size. |
Every term the collection defines is gathered in the glossary.