Seesaw Experimentation: A/B Tests with Spillovers

Abstract

This paper examines how a firm’s performance can decline despite consistently implementing successful A/B test innovations—a phenomenon we term “seesaw experimentation.” While these innovations improve the measured primary dimension, they create negative externalities in unmeasured secondary dimensions that exceed the gains. Using a multivariate normal distribution model, we identify the conditions for this decline and propose positive hurdle rates as a solution. Our analysis shows how to set optimal hurdle rates to best mitigate these negative externalities and provides practical guidance for experimental design by demonstrating how these rates should vary with underlying parameters.