Site Reliability Engineering

Chapter 27. Reliable Product Launches at Scale

Written by Rhandeev Singh and Sebastian Kirsch with Vivek Rau

Edited by Betsy Beyer

Internet companies like Google are able to launch new products and features in far more rapid iterations than traditional companies. Site Reliability’s role in this process is to enable a rapid pace of change without compromising stability of the site. We created a dedicated team of “Launch Coordination Engineers” to consult with engineering teams on the technical aspects of a successful launch.

The team also curated a “launch checklist” of common questions to ask about a launch, and recipes to solve common issues. The checklist proved to be a useful tool for ensuring reproducibly reliable launches.

Consider an ordinary Google service—for example, Keyhole, which serves satellite imagery for Google Maps and Google Earth. On a normal day, Keyhole serves up to several thousand satellite images per second. But on Christmas Eve in 2011, it received 25 times its normal peak traffic—upward of one million requests per second. What caused this massive surge in traffic?

Santa was coming.

A few years ago, Google collaborated with NORAD (the North American Aerospace Defense Command) to host a Christmas-themed website that tracked Santa’s progress around the world, allowing users to watch him deliver presents in real time. Part of the experience was a “virtual fly-over,” which used satellite imagery to track Santa’s progress over a simulated world.

While a project ...