Seesaw Experimentation: A/B Tests with Spillovers

Abstract

This paper investigates how a firm’s performance can decline even when consistently implementing innovations validated through A/B testing. We introduce the concept of seesaw experimentation, where successful innovations enhance performance in measured primary metrics but generate negative spillover effects in unmeasured secondary dimensions, ultimately reducing overall performance. We identify the conditions under which seesaw experimentation occurs. We also propose a simple solution to address it: implementing a positive hurdle rate for A/B tests. We derive the optimal hurdle rate for both the normal distribution and the fat-tailed distributions.