Author: Jin Li, Ye Luo and Xiaowei Zhang
| We show how a firm’s performance can decline despite consistently implementing successful A/B test innovations—a phenomenon we term “seesaw experimentation.” An improvement in the measured primary dimension can create negative externalities in unmeasured secondary dimensions that exceed the gains. Using a multivariate normal distribution model, we identify the conditions for this decline and propose optimal hurdle rates as a solution.