The Two Types of Statistical Errors

Most marketing decisions are made in the fog. Statistics is the flashlight—if you know where to aim it.

When you run an A/B test and declare a winner, you’re making a judgment under uncertainty. And like any judgment, it comes with risks: the risk of seeing a difference that isn’t there, or missing one that is. Enter Type I and Type II errors.

Rory Sutherland might call them the “tragedies of reason”—mistakes we make when we think we’re being rational. Deming would call them failures of the system.

Understanding these errors won’t eliminate risk. But it will help you avoid building a castle on noise.

Type I Error (False Positive)

You think you’ve found a signal—but it’s static. A Type I error occurs when you reject the null hypothesis (i.e., no difference) when it’s actually true.

In marketing, this often looks like:

“Version B increased conversions! Roll it out.”
Except it didn’t. You got lucky. Or the spike was due to seasonality, or an email blast, or a bug.

Consequences

• Wasted time scaling a false win
• Eroded trust in testing
• Complicated attribution in future tests

What contributes

• Too many simultaneous tests
• P-hacking and cherry-picking
• Low sample sizes

Type II Error (False Negative)

Here, the real effect is there—but you fail to see it. You keep the status quo, thinking the change didn’t work.

This often happens when teams give up too soon:

“We didn’t hit significance, so it must not matter.”
But maybe the test was underpowered. Maybe it did matter—you just blinked.

Consequences

• Missed opportunities
• Slower iteration cycles
• Undervalued innovation

What contributes

• Insufficient sample size
• High variability in user behavior
• Misaligned test duration

So... Which Is Worse?

That depends.
If you’re in healthcare or aerospace, Type I is deadly.
If you’re in marketing or product, the cost of a Type II error might be higher—missed growth, misunderstood users, invisible ROI.

Deming’s Take

Deming warned against tampering with systems based on noise. Mistaking common cause variation (inherent randomness) for special cause (something new and meaningful) leads to overcorrection and chaos.

In other words: don’t chase every fluctuation. Know your margin of error.

What You Can Do

• Pre-register your hypotheses. Know what you're testing and why.
• Use proper sample sizes. Don’t trust underpowered tests.
• Frame results in terms of risk. Ask: what's the cost if we’re wrong?
• Build an experimentation culture. Value learning over winning.

In Summary

Type I errors are false alarms.
Type II errors are missed calls.
Both can break a feedback loop if left unchecked.

The art of analytics isn’t just about measuring—it’s knowing what not to trust.
And that might be the most human thing about data.