Chapter 1. Defining Threat Intelligence
Threat intelligence is gaining a more prominent role in running a modern security team. Of course, this prominence means that every security professional and vendor also wants the world to adopt their vision of threat intelligence. This leaves many organizations with two questions: what is threat intelligence, and can it can really help improve security?
The short answer to the second question is: it can and does, when implemented correctly. But, as with any complex system, there is no “Easy Button” for threat intelligence. The goal of this book is to provide an introduction to some of the basic themes of threat intelligence. This book is not designed to be comprehensive; instead, it is designed to start a conversation about building a successful threat intelligence program. This book provides guidelines and exposes pitfalls for any organization that is ready to build a Threat Intelligence Unit for the first time, or is looking to improve their existing intelligence team.
This chapter starts by defining threat intelligence. As silly as this may sound, without a common definition of the term, it is hard to build an effective program. The rest of the book revolves around the definition and the basic tenets of threat intelligence defined in this chapter.
Military Terms
Threat intelligence in information security draws heavily upon years of intelligence experience from the military. Not just because the military has established intelligence frameworks, but because many information security threat intelligence professionals got their start in the military. To that end there are military intelligence terms used throughout this book, in part because these terms are commonly used by threat intelligence teams.
What Is Threat Intelligence?
There are a number of characteristics that define threat intelligence, but there is a general industry consensus around the definition proposed by Gartner:1
Threat intelligence is evidence-based knowledge, including context, mechanisms, indicators, implications and actionable advice, about an existing or emerging menace or hazard to assets that can be used to inform decisions regarding the subject’s response to that menace or hazard.
This definition covers all aspects of threat intelligence from collection, to processing, to the decision-making process. It also covers the many different types of information that should be collected as part of the intelligence process.
In short, threat intelligence is any information that can be correlated with additional information in a manner that allows an organization to improve their security in a tangible manner. For example, if a third-party vendor tells an organization that Anonymous has been launching attacks against other organizations in the same vertical, and provides a list of tools that Anonymous is using to launch these attacks, that is threat intelligence.
In order for data from outside sources to be considered threat intelligence, it must be:
- Relevant: It must impact the organization in some way.
- Actionable: Concrete steps can be taken by security teams to protect the organization.
- Contextual: There should be enough evidence included to enable an intelligence analyst to effectively rank the threat.
New threats spring up all the time, but not all of those threats are relevant to every organization. For example, a new vulnerability against the webmail application SquirrelMail is not relevant to an organization that is not running SquirrelMail, no matter how severe the vulnerability is.
Simply knowing that Anonymous has launched new attacks is not sufficient because there is no actionable information. That basic knowledge does not provide an organization with enough information to take steps to ensure it is protected against those attacks.
Finally, in order for information to be considered threat intelligence there must be context surrounding it. The SquirrelMail vulnerability mentioned above is a perfect example. While a vulnerability in a popular Internet-facing application could be a cause for concern, if an organization is properly managing and scanning its network assets, the security team should be able to quickly determine whether SquirrelMail is installed anywhere on the network. If SquirrelMail is not installed, then there is most likely no threat to the organization from the vulnerability. It is that additional context—knowledge of an organization’s network assets—that can turn information into threat intelligence.
Of course, context does not always have to originate from within an organization. In fact, context can be entirely external. For example, if a third party provides an organization with the email address wowsmith123456@posteo.net as an indicator associated with the infamous NotPetya ransomware attacks, that provides some level of context. If an organization matches against that address in their logs or in collected NetFlow data, they will know there is a good chance they have a NotPetya attack. The first part of the intelligence, that wowsmith123456@posteo.net is associated with the NotPetya ransomware attack, is not something that would be picked just reviewing internal data, unless the organization happened to fall victim to the NotPetya attack. Instead, the context around the email address would come from a third-party source that is collecting data from thousands of different networks.
Context can be deeper than just a single connection. Knowing that the email address is associated with the NotPetya attack is a baseline of useful information. But tying that email address to other indicators associated with the attack, such as IP addresses and domains used for command and control purposes or file hashes associated with the malware is even better. Providing information about attack atmospherics—such as what organizations are being targeted or the fact that the email address has been disabled, so if an organization is infected they should not try to pay the ransom—is the ideal level of context.
What Threat Intelligence Isn’t
Now that there is a consensus definition of threat intelligence, it is important to take a step back and explain what threat intelligence isn’t. Threat intelligence is not a list of IP addresses or domain names with no context. Threat intelligence is also not a proprietary platform that exists solely inside a security vendor’s tool. Finally, threat intelligence is not data that rests solely in a portal or in a report that is isolated from the rest of an organization’s network.
Each of the examples listed in the previous paragraph can help create threat intelligence, but they are not threat intelligence in and of themselves.
Examples of “not” threat intelligence that are seen most often tend to be around the context of the indicators. A third party might share that IP address 101.200.81.187 is malicious. That isn’t threat intelligence because there is no context surrounding it. Knowing that a third party classifies an IP address, domain, file hash, or some other indicator as malicious, without understanding how they came to that conclusion, is not threat intelligence. In fact, these types of context-free data feeds can make the security team’s job more difficult, as they don’t have any context around which to judge a threat.
Problems with context can even happen with internal intelligence, which is why documentation between teams is so important. For example, a firewall administrator may receive a report that several hosts on the network are attempting to call out to a single IP address several times a minute. Based on that report, the firewall administrator puts a rule into the firewall blocking traffic to that IP address and goes on to the next task.
With no documentation in place, when the firewall administrator goes home for the evening and the night administrator comes into work, there is no information about why the rule was created. In addition, there is no information about whether or not the original traffic was really a threat, or, if it was, what the associated malware was and whether it has been as successfully cleaned. Actions, even those seemingly benign as a firewall rule change, that pass between different security teams can become threat intelligence for the organization. But they require context to understand what the threat is and determine whether there are additional actions that need to be taken.
Data Cleansing
Data cleansing is the process of cleaning up data to remove obvious mistakes or incorrect entries and it is an important part of good threat intelligence. Intelligence delivered without at least some attempt at data cleansing is not threat intelligence, in fact it usually makes more work for intelligence teams and can make an organization less secure.
This is often seen in data feeds where RFC 1918 (the private IP space) addresses are accidentally slipped into the list or users are encouraged to block well-known IP addresses, such as 8.8.8.8 (one of Google’s DNS servers) with no explanation.
Even the best third-party providers occasionally make mistakes, but threat lists with no sanity check are not threat intelligence.
Data Feeds versus Threat Intelligence
Many organizations get their start with threat intelligence through data feeds. A data feed is a list of indicators provided by a third party that can be correlated against internal security systems to find matches that can be acted upon.
The appeal of data feeds is understandable—there is a lot of malicious activity happening at any given time on the Internet. No single organization can see all of that activity. By aggregating indicators from security incidents across the Internet and combining them into a single data feed, organizations can better protect themselves against coming attacks or use the indicators to find attacks they may have missed.
As most organizations quickly find out, data feeds rarely live up to their promise. Too often, data feeds contain outdated data and don’t provide context or give security teams enough information to make the data actionable. Even the purported purpose of data feed correlation, to provide relevance, doesn’t always work as expected.
Many organizations who start down the path of a threat intelligence program with data feeds are surprised to discover that these feeds often create more work than expected. Correlating data feeds against logs from other network devices in a Security Information and Event Manager (SIEM) seems like a natural fit—after all, it will help security teams identify security incidents that may have been missed otherwise. Unfortunately, what many security teams quickly find out is that, rather than making their lives easier, data feeds generate more work. By identifying even more security incidents and, often, many more false positives, data feeds can quickly become a burden to security teams.
That is the primary difference between a data feed and threat intelligence, even threat intelligence delivered in feed format. Threat intelligence helps security teams improve the security posture of the organization and makes the security team more effective and responsive to potential security incidents.
That doesn’t mean that data feeds don’t serve a purpose. In fact, the data feed concept is an important one for improving an organization’s security posture. One of the biggest challenges that organizations of all sizes face when it comes to network security is that for 30 years, the security industry, as a whole, has solved the latest security problem by “adding a box” to the mix. First, it was firewalls, then intrusion detection systems (IDS), proxies, endpoint protection, web application firewalls (WAF), data loss prevention (DLP), and so on, to the point that many organizations have 20 or more security systems in their network, none of which talk to each other.
This leads many security teams to suffer from “console fatigue.” They constantly have to jump from one security vendor’s console to another’s, and correlation is often done manually, as in “Oh wait, I think I saw that same indicator in an alert from my endpoint vendor. Let me go see if I can find it.”
Data feeds, delivered in an automated fashion, from one security system to another can help improve the security of an organization by having those systems communicate. That automation between systems, even cloud systems, allows the security team to have a better view of what is happening within their organization and allows them to make more effective and faster decisions when it comes to prioritizing incidents.
Even external data feeds, when they are processed correctly, can help an organization improve security. Some data feeds can be fed directly into a firewall, mail server, or even fed directly into a DNS server as a Response Policy Zone (RPZ). These tend to be specialty feeds that are well-vetted and serve a specific purpose. Most threat data feeds from reliable sources can contain valuable data that, when combined with other information, can eventually become threat intelligence.
Threat Intelligence from the Inside Out
Inevitably when an organization is serious about building out a threat intelligence program they start by looking externally. Whether the start involves looking for threat feeds, as described above, or reaching out to vendors who sell threat intelligence, there is a strong draw to rely on vendors to deliver threat intelligence.
The problem is that even the best threat intelligence providers can’t deliver effective threat intelligence unless the organization knows what their needs are. Remember, actual threat intelligence has to be relevant, provide context, and be actionable. In order for these requirements to be met there must be something against which to correlate the external collection.
To that end, the strongest threat intelligence programs start internally and work their way out. Often, this is the hardest part of building a threat intelligence program: getting the internal data from the network into the hands of the people who need to be able to analyze it.
Make no mistake, there is a lot of valuable security data inside an organization of any size that is owned by different teams and this data is very siloed. For example, the vulnerability management team gets threat intelligence from their vendors regarding new vulnerabilities and exploits that target those vulnerabilities. However, the vulnerability team is usually separate from the security team, so the security team doesn’t always learn about these new threats in a timely fashion. But it goes deeper than just getting data from one team to another—a successful threat intelligence program has to encompass the entire organization, and it needs to start from the top.
Defining a Mission
When starting a threat intelligence program, many organizations don’t bother answering the most fundamental question, “Why does our organization need threat intelligence?” In security engineering parlance this is known as the “What problem are we trying to solve?” question. Before doing anything else, this question must be answered in order for a threat intelligence program to be successful.
On the surface this may seem like an easy question to answer: “To protect our organization,” “To understand the threats facing our organization,” or “Because the CEO said to do it,” are all very common answers. And while those are important components of a threat intelligence program, they are not complete answers. Those are all surface answers (with the exception of the CEO answer), but they don’t really define the unique intelligence needs of an organization.
Instead, the goal of any threat intelligence program should be to protect the most valuable asset of an organization, the asset that if it wound up on a “paste site” would potentially irreparably damage the reputation or value of an organization. That asset could be a customer database, it could be the designs for new automobiles, it could be the proprietary formulas for beauty products. Whatever that asset is, it should be the core mission of a threat intelligence team to protect it; and all actions the team takes should derive from that mission. This is why it is so important to have senior management and the board of directors involved in creating a threat intelligence program, it helps to ensure the goals of the threat analysts align perfectly with the goals of the organization at large.
What Is a Paste Site?
A “paste site” is a website that is used to share plain text documents, usually anonymously, such as code. Some attackers use paste sites to post data they have collected from their attacks, especially if that data will be damaging or embarrassing to the victim organization. The most popular paste sites are Pastebin, Pastie, and Ghostbin, but there are dozens of others that are used by attackers.
It is not just attackers that use paste sites, there are a number of legitimate uses for them as well. Programmers will often paste large snippets of code to paste sites so that others can review the code. Many legitimate users think of paste sites as a safe place to store information, but the truth is that most of these sites are indexed and searchable both from the paste site’s search function and through larger search engines. Anything published on a paste site will most likely be viewable by anyone on the Internet.
That doesn’t mean that threat intelligence collection and analysis should only involve the core mission. What it does mean is that all intelligence requirements should stem from the core mission and support the core mission in some way. In other words, a threat intelligence program should start with a core mission and then ask the questions that will help support that mission.
One of the purposes of threat intelligence is to provide answers to questions that the leaders of an organization have. These questions are more rightfully referred to as intelligence requirements. There are two military definitions of intelligence requirements. The first is: Any subject, general or specific, upon which there is a need for the collection of information, or the production of intelligence. The second definition is: A requirement for intelligence to fill a gap in the command’s knowledge or understanding of the operational environment or threat forces.2
The first definition focuses on longer-term strategic intelligence, while the second definition is more immediate and revolves around tactical intelligence. Ultimately, in the realm of threat intelligence, the purpose of both types of intelligence requirements is to answer questions that leaders in the organization have about threats to the organization.
Understanding the Threat
Now that that the core mission has been defined, the threat intelligence team has to understand what the threats are to those assets which are vital to the mission. This may seem counterintuitive; after all, most people think that one of the purposes of threat intelligence is to inform organizations of threats. But that’s not exactly correct: good threat intelligence from a third party will inform an organization about specific threats, for example new vulnerabilities in a database or new techniques used to gain access to an organization, but it can’t determine what an organization sees as its threats. However, good third-party providers of threat intelligence will somewhat tailor intelligence to the specific needs of their customer (there are some caveats to this that will be discussed in Chapter 2).
To illustrate this idea more clearly let’s take the example of the customer database mentioned above. There are a number of potential threats against against a customer database, a few of these threats include:
- Vulnerabilities in the database software itself
- The risk of attackers accessing the data and selling it
- The risk of an employee accessing the database and taking that information to a competitor.
By first understanding and outlining the threats to the most valuable assets to an organization a threat intelligence team can start to build requirements that need to be satisfied both internally and externally.
Externally, the threat intelligence team can ask their providers for information about new vulnerabilities in the database software. They can also ask what are the most popular underground forums or carding sites where data similar to the organization’s is being sold, and if the provider will monitor those forums for mentions of the organization’s name. Even beyond that, the organization can ask which attack groups are targeting this type of data and what their tactics, techniques, and protocols (TTPs) are. Knowing the methods the attackers use might help an organization distinguish between an external attack and an insider threat.
These are just some examples of external-facing questions threat intelligence teams can ask. However, not all of the questions are external facing, there are also a lot of questions that the threat intelligence team needs to ask internally (more on this next).
Collecting the Data
Getting answers to the internal questions can sometimes be more difficult than getting answers to the external questions. After all, an organization pays security vendors and threat intelligence providers for that type of information, so they are incentivized to respond quickly and accurately. On the other hand, there are often years of rivalries between departments and groups within an organization. This can make data sharing difficult. But knowing the mission means that intelligence teams can focus on collecting the data needed to carry out that mission. That means having the full support of the organization is critical. Collecting the necessary data may require access to systems to which security teams may not have had previous access.
Being able to answer those internal questions is just as important as answering the external questions. In fact, it is necessary to have a thorough understanding of the internal functions of an organization before the data collected from external providers can become true threat intelligence.
Log collection should not be the first step in internal data collection. The first instinct many threat intelligence teams have when they think about data collection is log collection. At an operational level this makes sense—intelligence analysts want more data and want systems that can consume as much data as possible. Logs are an excellent source of data (log collection will be discussed in more detail in Chapter 2).
Instead, threat intelligence teams should start by thinking strategically within the organization. This builds on the knowledge collected during the process of understanding the threats and expanding outward to learn more about different business units within the organization and their processes. It is not just processes that need to be understood; intelligence teams must understand data flow throughout the organization. For example, knowing that programs A, B, and C feed into database X and that cloud service Z pulls from that database is also an important part of the assessment process. This threat assessment process should always have the goal of understanding the potential threats associated with these relevant organizational processes.
Breadth is an important part of any threat intelligence program. A threat intelligence team should always be looking to broaden their sources of information. Threat intelligence teams should strive to acquire data from as many sources and in as many forms as possible, which is why reaching out to other groups in the organization is so important.
Understanding the processes and workflow of the different teams in the organization helps the threat intelligence team develop a broad series of intelligence requirements. Some of those requirements will be shared with threat intelligence providers. However, many of those requirements will remain internal, and the intelligence team will be able to source the necessary information to satisfy those requirements entirely within data collected by the organization. Sometimes that source will be mined log data, for example. Whether it is from logs or other sources, this will all become part of the threat intelligence cycle discussed in the next chapter.
Summary
Threat intelligence is a complex topic that can be daunting to organizations that are just starting to build out a threat intelligence program. It doesn’t have to be though. Starting with the definition outlined in this chapter, it is possible to build out a threat intelligence program that uses relevant, actionable indicators to provide context to threat or a potential threat.
The ultimate purpose of threat intelligence and the threat intelligence team is to protect the core assets of an organization. In order to do that effectively, the threat intelligence team needs to understand the core mission of the organization and what the organization considers to be its most valuable data. All intelligence requirements should stem from that.
While third-party threat intelligence can be invaluable, it doesn’t help an organization improve its security unless it can be correlated against data collected within an organization. Some of that internally collected data will be log data, but internal intelligence is more than just log dat—it also involves understanding the processes of other groups within the organization.
1 McMillan, Rob, “Definition: Threat Intelligence”, Technology Research, May 16, 2013, accessed January 03, 2017.
2 Headquarters, Department of the Army. “Joint Intelligence (JP 2-0)”, 2013.
3 Weedon, Jen, William Nuland, and Alex Stamos, “Information Operations and Facebook”, Facebook, April 27, 2017, accessed 19 June 2017.
Get Threat Intelligence in Practice now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
