Threat Hunting

Chapter 1. Threat Hunting and Its Goals

In this chapter, we will discuss threat hunting, its goals, and its benefits. At the conclusion of this chapter, you should be able to explain what threat hunting is, where it came from, and how it relates to operational security.

What Threat Hunting Is

In the introduction, I said that “threat hunting is an advanced security analysis process that leverages deep knowledge of a network or organization to catch subtler, more deeply embedded attackers than a Security Operations Center (SOC) finds.” I want to unpack this statement a little to touch on several ideas: threat hunting as an advanced process, deep knowledge, the embedded attackers, and the relationship to the SOC.

Threat hunting is an advanced process: it is highly (but not entirely) self-directed and carried out by senior analysts. Good threat hunters are investigators, developers, teachers, and highly autodidactic. It is not easy for junior analysts to make the transition from a workflow-based, time-constrained lower-tier position to hunting, and when they do transition, they’ll make common mistakes such as chasing zebras.¹ You will cultivate your threat hunters from your junior staff; in the course of doing so you need to look for skills outside of tool mastery.

Threat hunting requires deep knowledge. The success of threat hunting processes goes hand in hand with a deep understanding of your network’s actual use—what services IT runs, what services shadow IT runs,² what’s exposed to the internet, what isn’t being monitored, and how to compensate for that. Most of your initial threat hunts will be turn into inventorying—figuring out what your network actually does. This also means that you can’t just hire a great threat hunter off the street; you need personnel who develop a deep understanding of your network and its operations.

Threat hunting is dealing with embedded attackers. This is, unfortunately, simply an acknowledgment that the attackers are really good at their job, and by the time we find out about the smart ones, they’ve been around for a while. One of the sea changes I see in information security in the past few years has been acknowledging that our fortress model doesn’t work. Threat hunting is born out of that acknowledgment: we must operate in a world where the enemy is already inside of our network.

Finally, threat hunting begins and ends with your SOC. Beyond simply culling the SOC for threat hunters, the success of the threat hunting process is in understanding weird phenomena and in some way turning that understanding into actionable results. The SOC uses those results, and any threat hunting team has to work smoothly with it.

Why Threat Hunting Matters

You need threat hunting because attackers and networks constantly evolve, necessitating that the defender be aware of these changes. To that end, I think there are four major reasons that you (and your C-suite) need to care about threat hunting: the changing landscape of attackers, the changing structure of networks, the power of your own understanding of your network, and the general advantage of better understanding your network.

Attackers change all the time, quelle surprise; anyone running enough of an operation to use threat intelligence (i.e., active SOC, threat intelligence cell processing raw data) has a keen awareness about the gap between when attackers change their tactics and when security catches up. Threat hunting gives defenders tools to reduce that gap—by actively looking for anomalous and suspicious behavior, defenders can identify changes in TTPs before they show up in the threat feed.

The other “inevitable” that defenders deal with are changes to their networks. Security controls are still based around the idea that the defender has a good understanding and control of their network. However, modern networks change their configuration and action on a regular basis. For security personnel, this is a constant struggle.

Complementing both of these problems are the security benefits from deeply understanding one’s network. Attackers have limited knowledge of a network’s structure and purpose; even insiders usually only have a good understanding of the systems that they directly access. Defenders can leverage knowledge of their network’s use, its composition and mission, to generate more effective defenses. To do so, they have to constantly chisel at the unknown features of their network, a process facilitated by threat hunting.

Finally, threat hunting provides an enterprise-wide benefit by enhancing the situational awareness of your network. The practices of effective threat hunting will yield insights about unknown systems, legacy technology, changes in network structure, and other phenomena that are not solely security concerns, but enterprise network management issues. Effective threat hunting will save time and money for the whole enterprise.

Why Now?

Threat hunting is not new,³ so why are people talking about it now? I contend that this because of how incident response has evolved in the last decade. Twenty years ago, threat hunting was basically what security people did. Networks were highly customized and managed internally, and half the job of a security person involved cobbling together tools to figure out what the heck was going on. Since that time, we’ve built up specialists, in the form of SOCs, but we haven’t liked doing that.

For years, the SOC has been the ugly duckling of information security—much of the work on analysis and SOC tools has been done with the goal of cutting SOC manpower via automation. However, in the last few years, there’s been an increased recognition that this is no more feasible than cities without police. Effective security demands effective SOC operations, and I see more tools that are focused less on supplanting analysts and more on empowering them. Threat hunting fits into this category; the biggest change for threat hunting isn’t the hunting process, it’s that there are now consumers.

Who Threat Hunting Is For: The SOCS

Figure 1-1 is a very high-level diagram of the duties of an ops floor; this breakdown consists of two teams: ops and data. The ops team consists of the people who directly respond to incidents, a hierarchy of analysts that begins with the intrusion detection system (IDS). The data team does not directly respond to incidents, instead providing support to the ops team by validating data, supplementing the data sources, and focusing on development and other support tasks.

Note the division in the ops side between the SOC and hunting; this division splits the workflow-based world of the SOC from the more open, loosely defined world of the hunt. The SOC is bounded by some kind of service level agreement (SLA) that defines the SOC’s responses, such as the time between an alert and action, and the types of actions available. SOCs have operational responsibilities that require some degree of predictability. The hunter will expand the SOC’s capabilities, but is engaged in a less well-defined, more iterative activity than the tight time bounds of the SOC can manage.

Threat hunters become advanced analysts by learning the network deeply. This is an important point, so I want to spell it out: threat hunters are made; they are most often made by practical SOC work. The most directly useful output from threat hunting work is information that improves the SOC’s effectiveness; examples of such output include:

• Identifying hostile tools, techniques, and procedures (TTPs)

• Mapping the network and the outside world

• New tools for detecting or classifying attacks

The data team supports the ops team by monitoring and managing data and analysis tools. The data team is explicitly removed from the immediate concerns of operations; this is necessary because ops is a very deadline-related job and operations personnel will get lost in the weeds. As shown in Figure 1-1, there are three particular roles that the data team has:

• The data team should manage the development and maintenance of analytics once prototypes have been developed. In this role, the data team will serve as an intermediary between the hunter and the SOC—the hunter generates prototypes, while the data team converts the prototypes into operational, optimized, and maintainable code.

• Threat intelligence. The data team is responsible for collecting, evaluating, normalizing, and presenting threat intelligence to both the hunters and the SOC. For the hunter the data team provides triggers that may initiate hunts (see Chapter 3 for more on this).

• Data integrity. The data team is responsible for ensuring that the data collected for operational use is valid, correctly formatted, and current. This task often overlaps with hunting; a common false positive leading to hunts is missing data, and it should be the data team’s responsibility to address why data is missing before a hunter has to. The data team will also work tightly with the hunter when pulling data for hunts.

Figure 1-1. Relating SOC to ops

A Concept: Analyst Attention

Imagine that in the course of a day, an analyst can process n events (the exact value of n is up for debate, but 100 seems to be a reasonable number). During their shift, they are going to receive some number that is much larger than n—because attacks are constantly increasing and our detection systems are a bit wonky.

I contend that the primary goal of the data team is to improve that n. We have several ways to do that, such as rolling events together so that if there’s a bunch of redundant events (attacks on the same address say), the analyst only has to deal with one super-event, or automating events so that the number of events the analyst has to pick and choose from is smaller.

The Threat Hunting Process as a Research Process

Figure 1-2 is a high-level summary of the threat hunting process; this diagram will guide the discussions of threat hunting throughout this report. As the figure shows, we break the hunting into three parts:

Trigger

This refers to the process by which a hunt begins. Threat hunts rarely begin from an IDS alert: instead they are driven by analyst intuition about something behaving odd on the network.

Hunt

The hunt is an iterative process by which analysts identify something odd about a network, test out hypotheses, and eventually come to a conclusion. As far as this report is concerned, the hunting process is in a logistical fight with all the other security processes going on at the same time; in particular, it’s contending for data access. To that end, the hunt is broken into two major loops—a big loop that describes the initial pull of the data, and a little loop that works with a data set pulled during the big loop.

Export

The conclusion of the hunt results in the hunter coming to a conclusion and communicating that result to the main operations team. Export may consist of software, reports, or new configurations for defense.

Figure 1-2. High-level threat hunting process

People who have experience with scientific research, rapid prototyping, or intelligence analysis will see similarities to those processes. However, as these processes all differ from each other, threat hunting also differs. In particular, I see threat hunting as having the following distinguishing characteristics:

Limited data

Threat hunting is generally conducted in situ; it takes place in response to a particular threat or incident, and as such it is working with data that is already affected by the incident in question. A scientist constructs an experiment trying to take advantage of the best collection they can, while the hunter starts with the data they have access to. (Attackers, regrettably, rarely contribute quality data sets to the community.)

Limited time

A threat hunter is operating within a constrained time frame. Threat hunting generally takes place in a business environment; there is a limited budget and it’s an expensive task.

Fewer universals

Threat hunting requires deep knowledge of an operational network, and as such requires information about a specific configuration. This information is hard to acquire, hard to learn, and hard to generalize.

Easier prototyping

Hunters generally have an easier time staging up experiments and prototypes because they have a clear idea of the operational environment they’re working with (their own).

We can borrow ideas from other disciplines to improve our work, in particular, mechanisms for determining if you’re wrong about something. To that end, I find that work on intelligence analysis and rapid prototyping is very helpful for threat hunters. Several of the books mentioned in Chapter 5 are useful for this purpose.

Conclusions

In this chapter, I have discussed the basics of threat hunting: why it exists, who it is for, and why it matters. I want you to understand three things at this point. First, threat hunting is not a new process; it’s an old process that has become a thing as our understanding of information security has evolved. Second, threat hunting is a process that exists to support the rest of your operational security—no SOC, no threat hunting. Finally, threat hunting is a research process; there are a lot of methods out there that are related to the scientific method, and threat hunting is in that family.

This context is important because threat hunting is a “formally young” field—by this, I mean that while threat hunting has gone on for years, it has been considered part of normal ops and learned mostly by doing. What’s changed in the last few years is that instead of just being a learn-by-doing activity, people have started to compare notes and identify what works, but it’s still early days. It’s good to read outside of the field (partly because there isn’t a lot you can read inside the field yet), and looking at how engineering and scientific processes work, how intelligence analysis and detectives operate, and work on rapid prototyping will help inform your hunts.

Now, let’s find out if you’re ready to hunt.

¹ Medical slang referring to the tendency of young doctors to assume the most exotic explanation for symptoms rather than the prosaic. If you hear hoofbeats behind you, think horses, not zebras. (Unless you live in Botswana; then you may want to turn around first.)

² The informal IT department in place because everybody hates the IT department; good threat hunters know shadow IT exists, great ones know their birthdays.

³ The first case I’ve put in the reading (Stoll) predates the fall of the Berlin Wall.