The article’s abstract is given below, with the full article available to read here., , . A fundamental problem of hypothesis testing with finite inventory in e-commerce. Appl Stochastic Models Bus Ind. 2021; 37: 454– 474. https://doi.org/10.1002/asmb.2574
We draw attention to a problem that is often overlooked or ignored by companies practicing hypothesis testing (A/B testing) in online environments. We show that conducting experiments on limited inventory that is shared between variants in the experiment can lead to high false-positive rates since the core assumption of independence between the groups is violated. We provide a detailed analysis of the problem in a simplified setting whose parameters are informed by realistic scenarios. The setting we consider is a two-dimensional (2D) random walk in a semiinfinite strip. It is rich enough to take a finite inventory into account, but is at the same time simple enough to allow for a closed form of the false-positive probability. We prove that high false-positive rates can occur, and develop tools that are suitable to help design adequate tests in follow-up work. Our results also show that high false-negative rates may occur. The proofs rely on a functional limit theorem for the 2D random walk in a semiinfinite strip.