The Dead Salmon Problem in AB Testing - Robin Fisher

There’s no shortage of marketing and promotion ideas for increasing revenue. But which are the best? To answer this question, we often turn to AB testing. Try out idea A on one group of consumers and idea B on another group, measure the effects, run some statistics, and voila, we have our answer. 

​Or do we? 

​In 2009, a group of neuroscientists did an AB study of their own. They tested whether the brain of a fish, would respond to emotionally charged images of humans. They used functional magnetic resonance imaging (fMRI) a widely used tool that measures brain activity by looking for changes in the MRI signal during Task A (say, deciding what emotion is depicted in each of a series of human pictures) compared to Task B (say, looking at a blank screen). Brain regions that have a stronger signal in A compared to B, are said to “activate” to A and, accordingly, are assumed to play a role in Task A performance. 

The study team did ​​exactly this AB test in their fish subject and, after running the standard statistical analyses, found a region of the fish’s brain that activated to human emotional depictions. The problem is that the result couldn’t possibly be true: the fish was a dead salmon purchased at a local grocery store.  The apparent “activation” wasn’t real at all. Instead resulted from an AB design statistical pitfall. The headlines read, “Scanning Dead Salmon in fMRI Machine Highlights Risk of Red Herrings.”

​The same pitfalls exist in consumer AB testing. Partnering with a large clothing retailer, we ran several “dead salmon” tests using multi-year e-com data. Our analysis showed that statistically significant false positives (like the dead salmon’s “activated” brain) occurred in up to 40% of AB tests. This means that if you rely on standard AB test designs and statistics, you have likely made important decisions based on inaccurate conclusions.
 
Neuroscientists have learned from the dead salmon study and devised ways to avoid AB test red herrings. These methods can be applied to retail too. We found that just a few tweaks can drop the erroneous conclusion rate down to near zero. But first, we need to recognize that a problem exists.