SRE Best Practices: Implementing Automation to Reduce Toil

Homeowners understand the unexpected amount of work it takes to keep a house running smoothly. For site reliability engineers, our home is our production environment. As systems and services scale, the amount of work it takes to just keep them running tends to scale up too. In SRE, this work—repeatable and not requiring human judgment—is known as toil. It’s the kind of work necessary for keeping the lights on that doesn’t meaningfully improve the services we’re responsible for. However, it’s critical for the health and success of SRE teams that toil is managed effectively.

The primary tool for eliminating or reducing toil is automation. But automation is a tricky beast. Get overaggressive and it can become the culprit of incidents and overall pain. Ignore it and your team will drown in work without being able to complete projects. The solution is to take a mindful approach, starting at checklists and moving gradually to fully automated services. That journey isn’t easy, but it’s the key to developing an effective SRE team.

Join Incident Labs’ Emil Stolarsky and Jaime Woo to learn how to identify toil and manage it effectively. You’ll explore the techniques companies like Google, Facebook, and Microsoft use for managing their toil with automation as well as strategies for bringing these ideas back to your organization to unlock time for your team.