Sampling & Confidence Intervals

Everything so far quietly assumed you knew the true mean and standard deviation of whatever you were studying. In reality you almost never do. You cannot measure every stock trade in history or survey every investor. Instead you take a sample, a smaller collection of observations, and use it to estimate the truth about the whole population. This lesson is about doing that honestly and knowing how much to trust the result.

Population, sample, parameter, statistic

A quick vocabulary refresher, because the distinction is the heart of this lesson. A population is the entire group you care about, and a sample is the subset you actually observe. A number that describes the whole population, like its true mean, is called a parameter, and it is usually unknown. A number you compute from your sample, like the sample mean, is called a statistic, and you use it as your best estimate of the parameter. Statistics is largely the discipline of using known sample statistics to reason about unknown population parameters.

Sampling variability

Here is the key insight. If you take a sample and compute its mean, then take a different sample and compute its mean, the two will not be identical. Each sample is a slightly different draw from the population, so each sample mean wobbles around the true mean by chance. This is called sampling variability, and it is not a mistake. It is a fact of working with samples. The crucial question becomes how much your sample estimate is likely to wobble, because that tells you how much to trust it.

The central limit theorem and standard error

A remarkable result called the central limit theorem says that if you took many samples and plotted all their means, those means would form a normal, bell-shaped distribution centered on the true population mean, even when the underlying data is not itself normal. The spread of that distribution of sample means is called the standard error, and it shrinks as the sample gets larger. This is why bigger samples give more reliable estimates: their sample means cluster more tightly around the truth. A sample of 1,000 pins down the mean far better than a sample of 10.

Confidence intervals

Because a single sample mean is just an estimate, reporting it alone is misleading. It is more honest to report a range that likely contains the true value, and that range is a confidence interval. A common choice is the 95 percent confidence interval, built roughly as the sample mean plus or minus about two standard errors. The interval expresses your uncertainty: a wide interval means the estimate is shaky, and a narrow one means it is precise. Larger samples, with their smaller standard errors, produce narrower, more useful intervals.

The formula in symbols

The interval runs from two standard errors below to two standard errors above the sample mean. That 2 comes from the empirical rule, since about 95 percent of a normal distribution lies within two standard deviations of the center. The worked example below builds one from a mean and a standard error.

What a confidence interval does NOT mean

This is the subtle part that trips people up, so read it slowly. A 95 percent confidence interval does not mean there is a 95 percent probability that the true value lies inside this particular interval. The true value is fixed. It is either in your interval or it is not. What the 95 percent actually refers to is the method: if you repeated the whole sampling process many times and built an interval each time, about 95 percent of those intervals would contain the true value. It is a statement about the reliability of the procedure, not about your one specific interval. This is exactly the kind of distinction ordinary statistics courses gloss over.

A confidence interval is a promise about the method, not about the single interval in front of you. The truth does not move. Your interval is just one of many the procedure could have produced.

Garbage in, garbage out: sampling bias

None of this machinery helps if the sample itself is not representative of the population. A biased sample gives a confident, precise, and wrong answer. The best-known example in finance is survivorship bias: measuring the performance of funds or stocks that still exist today, while silently ignoring all the ones that failed and disappeared. That makes the past look far safer and more profitable than it truly was. A related trap is selection bias, where the way the sample is gathered skews it from the start. Before trusting any estimate or confidence interval, the honest analyst asks whether the sample fairly represents the whole, and what might be missing from it.

You can now estimate an unknown value from a sample and honestly express your uncertainty. The next lesson uses these ideas to answer a sharper question: is an observed result real, or could it easily be luck? That is hypothesis testing and the p-value.