The Look-Elsewhere Effect: Why Your 'Significant' Finding Is Probably a Trick of the Search Space

Particle physicists have a rule that most data scientists have never heard of, and it would save a lot of careers if more people knew it.

Photo by Negative Space on Pexels.

In 2011, before the Higgs boson was officially confirmed, researchers at CERN kept finding tantalizing bumps in their collision data. Exciting stuff, except some of those bumps turned out to be nothing. The problem wasn't fraud or sloppy math. It was the look-elsewhere effect: when you scan a wide range of possible locations for a signal, you will find a bump somewhere just by chance, even if there's no real signal anywhere.

Here's the intuition. Imagine you're hunting for a single significant spike in a smooth curve of noise. If you only check one specific point on that curve, the odds of a false alarm are small, maybe 1 in 20 at the standard p < 0.05 threshold. But if you scan a thousand points along that curve looking for the highest one, you're not asking "is this point significant?" You're asking "is the best of a thousand points significant?" Those are wildly different questions, and the second one is almost guaranteed to say yes somewhere.

This is a cousin of the multiple comparisons problem, but it's sneakier because the comparisons aren't discrete. You're not testing five separate hypotheses, you're scanning a continuous range (a mass, a wavelength, a date, a dosage level) and asking where along that range something interesting pops up. The number of effective "tests" you're running is harder to count, which makes the correction harder to apply and easier to skip.

Where this shows up outside physics

You don't need a particle accelerator to fall into this trap. A few places it hides in ordinary data work:

• Time series anomaly detection. Scan five years of daily sales data for the single weirdest day, and you'll find one. That doesn't mean something happened. It means you looked at 1,825 days and picked the extreme.
• Genomics. Scanning the entire genome for a location associated with a disease is a textbook look-elsewhere problem. This is why genome-wide association studies use brutally strict significance thresholds (often p < 5 × 10⁻⁸) instead of the usual 0.05.
• Marketing segment hunting. Slice your user base by every combination of age, region, device, and signup channel, and some slice will show a huge lift. It's not a strategy. It's the search space talking.
• Financial backtesting. Try enough trading rules against historical prices and one of them will have crushed the market in hindsight. The rule didn't find alpha. You found noise with enough attempts.

graph TD
    A[Scan wide range for a signal] --> B{How many positions checked?}
    B -->|Few, pre-specified| C[Standard p-value roughly valid]
    B -->|Many, unspecified| D[Effective number of tests balloons]
    D --> E[Correct threshold or report global p-value]
    E --> F(Trust the finding)
    D --> G[Skip correction]
    G --> H((False discovery))

What to actually do about it

The physics community's answer is to report both a "local" p-value (how surprising this specific bump is, assuming you knew to look there) and a "global" p-value (how surprising it is that a bump this size showed up anywhere in the range you searched). The gap between the two numbers is often enormous, sometimes by several orders of magnitude.

For everyday data work, you won't always have the machinery to compute an exact global p-value, but you can still protect yourself. Pre-register which comparisons you actually care about before you see the data. If you're doing exploratory scanning, say so explicitly in your writeup and treat anything you find as a hypothesis to test on fresh data, not a conclusion. Holdout sets exist for exactly this reason: the bump you found in your training window needs to survive contact with data it hasn't seen.

The uncomfortable truth is that "we scanned until we found something" is the default mode of most exploratory analysis, and almost nobody labels it that way. The bump in your dashboard is real in the sense that the number exists. Whether it means anything depends entirely on how hard you had to search to find it, and that number rarely makes it into the slide deck.