The Dark Side of A/B Testing – Avoiding the Pitfalls

As product designers and marketers, we love the clarity that comes from A/B testing. We have an idea; we implement it; then, we let the customers decide whether it’s good or not by running a controlled experiment. We split the users into unbiased groups and watch one treatment overtake the other in a statistically significant, unbiased sample.

Thoroughly impressed by our own analytical rigor, we then scale up the winning treatment and move on. You guessed it: I’m about to poke holes in one of the most sacred practices in tech, A/B testing.

Continue reading
Advertisements