Null Hypothesis Significance Testing: The Framework That Answers a Question Nobody Asked

Somewhere in a research lab right now, someone is staring at a p-value of 0.048 and feeling relieved. They shouldn't be. What that number tells them is not what they think it tells them, and the gap between the two things has quietly corrupted decades of published science.

Photo by Tara Winstead on Pexels.

Null Hypothesis Significance Testing (NHST) is the dominant statistical procedure in medicine, psychology, economics, and most of social science. The basic ritual goes like this: assume the null hypothesis is true (usually "there is no effect"), collect data, calculate how often you'd see data this extreme under that assumption, and if that probability falls below 0.05, declare victory and write the paper.

The problem is fundamental. NHST calculates P(data | null hypothesis is true). Researchers want to know P(null hypothesis is true | data). These are not the same number. Treating them as interchangeable is a logical error with a name: transposing the conditional. It's the same mistake as concluding that because most lottery winners bought a ticket, most ticket buyers will win.

So what does p = 0.03 actually mean? It means: if the null hypothesis were true and you repeated this experiment many times, you'd see results at least this extreme about 3% of the time. That's it. The p-value says nothing about whether your hypothesis is true, nothing about effect size, nothing about whether the result will replicate. It measures the surprise value of your data under a hypothetical world you probably don't live in.

This distinction isn't academic.

Consider a drug trial. A p-value of 0.04 gets the drug published. But the actual probability that the drug works depends on the prior probability that any randomly selected drug candidate is effective (historically quite low, somewhere around 10% for most disease areas), the statistical power of the study, and the false positive rate. Run those numbers through Bayes' theorem and a "significant" result from a single underpowered trial might carry only a 20-30% chance of reflecting a real effect. The math is in Ioannidis's famous 2005 paper, and the conclusion is uncomfortable: most published research findings are probably false, not because researchers cheat, but because the procedure is miscalibrated for the questions being asked.

The 0.05 threshold makes everything worse. It's arbitrary (Fisher himself never intended it as a hard cutoff), it's binary (0.049 is "significant," 0.051 is not), and it creates perverse incentives. When careers depend on crossing a threshold, people will cross it by any means available: collecting data until the p-value cooperates, testing multiple outcomes and reporting the winner, quietly dropping participants who make the results messy. None of this requires conscious dishonesty. The structure of the incentive does the work.

Here's a cleaner picture of what the procedure actually tests versus what researchers want to know:

graph TD
    A[Collect Data] --> B{Calculate p-value}
    B --> C[P(data | H0 true)]
    B --> D[P(H0 true | data)]
    C --> E(What NHST gives you)
    D --> F(What researchers want)
    E --> G{Are these equal?}
    F --> G
    G --> H[No. Requires prior probability.]

The fix people reach for is usually Bayesian inference, which directly estimates the probability of the hypothesis given the data (using a prior). That's a real improvement, though it introduces its own arguments about where priors come from. A more modest fix is to report effect sizes and confidence intervals alongside p-values, which at least forces a conversation about whether an effect is large enough to care about. A drug that reduces systolic blood pressure by 1 mmHg with p = 0.001 is statistically significant and clinically useless.

Some journals have started banning p-values outright. The American Statistical Association issued statements in 2016 and 2019 walking back the tyranny of 0.05 with unusual directness for a professional body. Neither move has changed the culture much. The ritual is too entrenched, too easy to teach, and too convenient for anyone who needs a clean answer.

What should you do when you read a study? Check the effect size first. A tiny effect with a tiny p-value usually means a large sample was used to detect something that barely matters. Ask whether the study was pre-registered (did the researchers commit to their hypothesis and analysis plan before seeing the data?). Look for replication. A single significant result is weak evidence of almost anything.

The p-value isn't wrong as a calculation. It's wrong as an answer. It computes something precise and then gets used to conclude something different. That's a quiet disaster dressed up in respectable notation, and it's been running the show for nearly a century.