There’s no shortage of marketing and promotion ideas for increasing revenue. But which are the best? To answer this question, we often turn to AB testing. Try out idea A on one group of consumers and idea B on another group, measure the effects, run some statistics, and voila, we have our answer.
Or do we?
The study team did exactly this AB test in their fish subject and, after running the standard statistical analyses, found a region of the fish’s brain that activated to human emotional depictions. The problem is that the result couldn’t possibly be true: the fish was a dead salmon purchased at a local grocery store. The apparent “activation” wasn’t real at all. Instead resulted from an AB design statistical pitfall. The headlines read, “Scanning Dead Salmon in fMRI Machine Highlights Risk of Red Herrings.”
The same pitfalls exist in consumer AB testing. Partnering with a large clothing retailer, we ran several “dead salmon” tests using multi-year e-com data. Our analysis showed that statistically significant false positives (like the dead salmon’s “activated” brain) occurred in up to 40% of AB tests. This means that if you rely on standard AB test designs and statistics, you have likely made important decisions based on inaccurate conclusions.
Neuroscientists have learned from the dead salmon study and devised ways to avoid AB test red herrings. These methods can be applied to retail too. We found that just a few tweaks can drop the erroneous conclusion rate down to near zero. But first, we need to recognize that a problem exists.