Bayesian Methods for Hackers: Probabilistic Programming and Bayesian Inference

7. Bayesian A/B Testing

7.1 Introduction

Part of a statistician’s or data scientist’s goal is to be a champion of experiments, and one of the best tools for a data scientist is a well-designed split-test experiment. We’ve seen split-tests previously. In Chapter 2, we introduced Bayesian analysis of an A/B test on conversion rates for a Web site. This chapter will extend that analysis to new areas.

7.2 Conversion Testing Recap

The fundamental idea in an A/B test is that we consider a perfect counterfactual universe, where the population under study is identical but subject to some treatment, then any differences between the populations after the study must be attributed to the treatment. In practice, we can’t spin up other universes, so we ...