Post-Incident Reviews

Chapter 6. Outage: A Case Study Examining the Unique Phases of an Incident

Day One

Detection

Around 4 p.m. Gary, a member of the support team for a growing company, begins receiving notifications from Twitter that the company is being mentioned more than usual. After wrapping up responding to a few support cases, Gary logs into Twitter and sees that several users are complaining that they are not able to access the service’s login page.

Gary then reaches out to Cathy, who happens to be the first engineer he sees online and logged into the company chat tool. She says she’ll take a look and reach out to others on the team if she can’t figure out what’s going on and fix it. Gary then files a ticket in the customer support system for follow-up and reporting.

Note that Gary was the first to know of the problem internally, but that external users or customers first detected the disruption and a sense of urgency did not set in until Twitter notifications became alarmingly frequent. Still, responding to support cases was of higher priority to Gary at that moment, extending the elapsed time of the detection phase of the incident.

How many ideas for improvement can you spot in this scenario?

Response

Cathy attempts to verify the complaint by accessing the login page herself. Sure enough, it’s throwing an error. She then proceeds to figure out which systems are affected and how to get access to them. After several minutes of searching her inbox she locates a Google Document explaining ...