Simpson's Paradox: When Your Data Tells Two Opposite Stories

Your company's overall performance metrics look terrible. Sales are down across every region, customer satisfaction has plummeted, and conversion rates are embarrassing. But here's the kicker: when you drill down into each individual market segment, performance has actually improved everywhere.

Photo by Ann H on Pexels.

Welcome to Simpson's Paradox—the statistical minefield that makes your data lie with mathematical precision.

This isn't some obscure academic curiosity. Simpson's Paradox shows up in business dashboards, medical studies, and policy decisions with disturbing regularity. The trend you see in your aggregated data completely reverses when you break it down by subgroups. And vice versa.

The Berkeley Gender Bias That Wasn't

UC Berkeley got sued in 1973 for gender discrimination in graduate admissions. The numbers seemed damning: men were accepted at a 44% rate while women faced a 35% acceptance rate. Clear evidence of bias, right?

Wrong. When researchers examined individual departments, they found something shocking. Most departments actually showed a slight bias toward women. The paradox emerged because women disproportionately applied to highly competitive departments with low acceptance rates across the board.

The aggregated data told one story. The disaggregated data told the opposite. Both were mathematically correct.

How This Sabotages Your Analysis

Simpson's Paradox strikes when there's a lurking variable—some hidden factor that confounds your interpretation. The variable splits your data into subgroups with different underlying characteristics.

graph TD
    A[Raw Data] --> B{Hidden Variable}
    B --> C[Subgroup 1]
    B --> D[Subgroup 2]
    B --> E[Subgroup 3]
    A --> F[Aggregate Analysis]
    C --> G[Disaggregate Analysis]
    D --> G
    E --> G
    F --> H[Conclusion A]
    G --> I[Conclusion B]

Your marketing team celebrates improved click-through rates across all campaigns. But when you segment by traffic source, organic search is tanking while paid ads are crushing it. The overall improvement? Just a shift in traffic mix, not campaign performance.

Your product team sees user engagement dropping month over month. Panic sets in. But split by user cohorts, and you'll find that engagement within each cohort is actually rising. New users are just starting from a lower baseline—and you're acquiring more of them.

Where Simpson's Paradox Hides

This paradox loves to lurk in time-series data. Economic indicators flip when adjusted for demographic changes. Stock market correlations reverse when segmented by market cap. Website conversion rates tell contradictory stories when split by device type or user acquisition channel.

Medical research gets hammered by this constantly. Drug treatments appear ineffective in aggregate but show clear benefits within age groups. Surgical outcomes look worse at teaching hospitals—until you account for case complexity.

The scariest part? You might never know it's happening. Most analysis stops at the aggregate level. The paradox stays hidden unless someone thinks to slice the data differently.

Defending Against the Paradox

First rule: always stratify your analysis. Don't just look at overall trends—break them down by relevant subgroups. Customer segments, time periods, geographic regions, product lines. The lurking variable could be anywhere.

Second: understand your data generating process. How does your sample get composed? What factors influence both your outcome variable and the likelihood of being included? In Berkeley's case, department choice influenced both admission odds and gender representation.

Third: be suspicious of dramatic changes in trends, especially when accompanied by shifts in sample composition. If your user base is evolving, your metrics will reflect that mix change, not just underlying performance.

The Real Lesson

Simpson's Paradox isn't just a statistical curiosity—it's a reminder that correlation can be utterly meaningless without proper context. Your data isn't lying, but your interpretation might be.

The next time someone presents you with clean, aggregated metrics that tell a simple story, ask the uncomfortable question: what happens when we break this down? The answer might flip everything you thought you knew.

Because sometimes the most dangerous insights are the ones that look obviously true.