Hypothesis Testing & P-Values

How do you decide whether a result is real or just luck? This is a central question in quantitative finance. A strategy made money in a backtest, but was that genuine skill or a lucky roll of the dice? Hypothesis testing is the formal framework for answering this, and the p-value is the number that summarizes the answer. These are among the most used, and most misused, tools in statistics, so we will go slowly.

The two hypotheses

• The null hypothesis is the skeptical default: there is no effect, no edge, nothing going on. For a trading strategy, the null says the strategy has no real edge and its apparent returns are just chance.
• The alternative hypothesis is the claim you are testing for: there is a real effect. For the strategy, the alternative says it truly does have an edge.

The logic of the test

The core move is a little backwards, so read it carefully. You start by assuming the null hypothesis is true, that there is no effect. Then you ask: if there really were no effect, how likely would it be to see data as extreme as what I actually observed? If your data would be very unlikely under the null, that is evidence against the null, and you reject it in favor of the alternative. If your data is reasonably ordinary under the null, you fail to reject it, meaning you have not found convincing evidence of an effect. It works like a court presuming innocence: you assume no effect and demand strong evidence before overturning that assumption.

You can only reject, never prove

A crucial subtlety: hypothesis testing never proves the alternative. It can only find evidence against the null. When you reject the null, you are saying the data is hard to explain by chance alone, which supports the alternative, but does not prove it. And when you fail to reject the null, you have not proven there is no effect, only that you lack enough evidence to claim one. This mirrors the scientific method, where theories are supported or falsified but never finally proven, and it instills the right humility about what data can establish.

The p-value

The p-value puts a number on all this. It is the probability of observing data at least as extreme as what you actually saw, assuming the null hypothesis is true. A small p-value means your data would be very unlikely if there were no effect, which is evidence against the null. A large p-value means your data is quite consistent with the null, so there is little evidence of an effect. By common convention, a p-value below 0.05, that is 5 percent, is called statistically significant and taken as grounds to reject the null. That 5 percent threshold is just a convention, not a law of nature.

What a p-value is NOT

This is where fortunes and reputations have been lost, so slow down here. Three misconceptions to clear up.

• A p-value is not the probability that the null hypothesis is true. It is calculated by assuming the null is true, so it cannot also tell you the chance the null is true. That would be circular.
• A p-value is not the probability your result happened by chance. It is a statement about the data under a specific assumption, not a measure of how likely your finding is to be a fluke.
• A p-value says nothing about the size or importance of an effect. A result can be highly statistically significant yet trivially small in the real world. Statistical significance and practical significance are entirely different things.

The p-hacking trap

Here is the danger most relevant to finance, and it connects to overfitting. If you test many hypotheses against the same data, some will look statistically significant by pure luck. With a 5 percent threshold, roughly one in twenty completely random tests will cross the line by chance alone. Searching through many ideas until something passes, then reporting only that winner, is called p-hacking or data dredging. In finance, where it is easy to test thousands of strategies and indicators against history, this is an enormous hazard. The strategy that looks significant might simply be the lucky one out of hundreds you tried, with no real predictive power. The honest fix is to specify your hypothesis in advance and to count, out loud, how many things you tested before finding the significant one.

Test twenty random ideas and one will look significant by luck. The p-value of the survivor is meaningless if you never count the failures you threw away.

You now have the core inference toolkit: estimation, confidence intervals, and hypothesis testing. From here the unit turns to putting statistics to work in finance, starting with the idea that makes diversification the closest thing to a free lunch.