Incident Response

Fire Drills

By now, the plans for your team should be taking shape, as you have considered planning, staffing, documenting, and marketing your team’s services, policies, processes, and procedures. At some point (hopefully before a real emergency occurs) it will be beneficial to conduct some fire drills to test your team’s capabilities. Going back to the fire department analogy, the fire chief doesn’t send all his resources, trucks and staff, out the door for each alarm; rather, a specific group of resources is deployed depending on the type of emergency.

One thing to bear in mind is that drill is never a substitute for the real thing; it’s easy to train and drill, but until the team is actually battle-tested, you’ll never really know how it responds under real pressure. The two primary goals of training are to foster effective operational teamwork, and to transform written processes and procedures into instinctive actions for the team so that during a real crisis, effort is focused on the problem at hand, not on applying the basics.

Nonetheless, a good training program is important for the team as well as for the nonteam players we discuss earlier in this chapter. The most common types of fire drills are role-playing and live drills. In role-playing drills, the team -- along with any other drill participants -- sits in a room and acts out a realistic scenario and its associated response. In a live drill, however, the team’s procedures are tested, with few of the participants aware that it is a drill.

The main goal of a role-playing exercise is to test the team’s policies from a theoretical standpoint. The session should present a realistic incident scenario in the form of events that occur on a real timeline. The participants discuss the actions they took, their roles and responsibilities, and their motivators and sensitivities regarding the actions. When conducted well, a role-playing exercise can be an enormously useful tool for illustrating the team’s strengths and weaknesses, and for finding holes in the SOPs.

As professional incident responders, we have both been involved in numerous role-playing drills and have seen them work very effectively. Most of the better ones we’ve seen had one or two rooms full of the key players. In some cases, two groups of participants were chosen: a technical group consisting of principally IT staff, and an executive group made up of a team of the company’s senior executives. The most interesting outcome from the two group sessions was frequently the vast chasm that existed between the executives and the techies. On the other hand, we’ve also seen several highly effective role-playing sessions where all participants were in one room. Early on, we learned that having an outside expert assist in drafting and facilitating the drill is very helpful -- not only is such a person unattached to the organization or company’s politics or thinking, but can serve as a referee if there is a problem among the participants.

The point is that a role-playing exercise can be accomplished in a number of ways. Each method has its own pros and cons, with the value of the exercise being what the participants make of it and learn from it. When done well, it can be challenging, educational, and fun. If done poorly, or with less than complete commitment to the endeavor, it can be a waste of everyone’s time. A critical determining factor is to set each participants’ expectations properly prior to the exercise. Be sure to explain carefully to all of the participants what they will be expected to do and what they should get out of the session.

The other kind of fire drill is a live drill. In the ideal live drill, some of the participants won’t be aware that it is actually a drill. This type of drill follows the traditional fire drill that many people are familiar with, such as when the fire alarm goes off and everyone evacuates the building, not knowing whether it is a drill or a real fire. The incident response analogy is to have someone, whether an employee or a hired gun, trigger an alarm by, for example, attempting to break into one of the company’s firewalls. If (hopefully “when”) the system operations staff notices the alarm, the drill should capture their actions and record what takes place. Naturally, there needs to be some oversight and control in the process so that the exercise stops one step short of actually calling in the cavalry and waking up the CEO or releasing press statements to the public.

This type of live exercise is commonly mistaken for a network security penetration test. The difference, however, between a live incident fire drill and a penetration test is in what is being measured. An incident fire drill seeks to capture and measure the responses of the people involved in handling the staged incident, while a penetration test is intended to measure, and possibly exploit, system and network vulnerabilities. The purposes are fundamentally different, and the outcomes should be equally different. One measures actions, the other, vulnerabilities.

In any event, a live drill needs to capture and measure the responses of the people involved. The purpose of this, of course, is to analyze those responses (a role-playing exercise should also be carefully logged and recorded for the same reason). Any exercise or drill, live or otherwise, should be run for the purpose of improving the incident response procedures. It is often the case, for example, that some of the company’s policies are discovered to be either unenforceable or unfeasible when examined under the magnifying glass of an incident exercise.

In order to get the most out of an incident situation -- real or simulated -- a candid and frank postmortem review should be held as soon as possible after the event. In the postmortem session, all of the participants should be encouraged to speak honestly, without fear of reprisal, and critique the company’s incident response plan and the effectiveness of the simulation itself. Ideally, an independent third party should keep the notes on the postmortem session and make them available to the participants, highlighting the lessons learned by and comments of the participants, both positive and constructive.

From the issues raised in the postmortem, the company’s incident response policies and procedures should be improved wherever possible. Equally important is to pass on any changes to the people who need to be aware of them. By repeatedly using this process of plan, test, measure, analyze, and optimize, the company will be able to set up and maintain an incident response program that meets and exceeds the needs of the company.