Experiment Rigor for the Design and Analysis of Switchback Experiments

Yimin Yi | Senior Data Scientist ‪@Shipt‬
Experiment Rigor for the Design and Analysis of Switchback Experiments
Experiments in a three-sided platform like Shipt are challenging as we connect customers, shoppers, and retailers. The interconnection feature of our platform invalidates some experiments that randomize customers or shoppers to treatment and control. For example, an experiment on some algorithm change that intends only to impact shoppers in the treatment group may also affect shoppers in the control group through the interconnection of shoppers and customers. To overcome this limitation, we run switchback experiments in which we switch back and forth treatment and control status in some geographical regions over time. Running switchback experiments needs to answer some critical questions, such as deciding how long a switchback experiment should run and how to properly analyze switchback experiment data for statistical inference. This talk will provide some insights to answer these questions.