CHAPTER 3
How to Experiment Online
It doesn’t make any difference how beautiful your guess is, it doesn’t make any difference how smart you are, who made the guess, or what his name is. If it disagrees with experiment, it’s wrong. That’s all there is to it.
—RICHARD FEYNMAN, PHYSICIST, TEACHER, AND STORYTELLER
In 2012, a Microsoft employee working on its Bing search engine had an idea about changing the way it displayed ad headlines.1 The idea seemed trivial: add some of the ad’s subtext to the headline to make it longer (see figure 3-1). Developing the change wouldn’t require much effort—just a few days of an engineer’s time—but it was one of hundreds of ideas proposed, and the program managers deemed it a low priority. So it languished for ...
Get Experimentation Works now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
