The security data lake

Time-Series Data

The majority of security data falls into the category of time-series data, or log data. These logs are mostly single-line records containing a timestamp. Common examples come from firewalls, intrusion-detection systems, antivirus software, operating systems, proxies, and web servers. In some contexts, these logs are also called events, or alerts. Sometimes metrics or even transactions are communicated in log data.

Some data comes in binary form, which is harder to manage than textual logs. Packet captures (PCAPs) are one such source. This data source has slightly different requirements in the context of a data lake. Specifically because of its volume and complexity, we need clever ways of dealing with PCAPs (for further discussion of PCAPs, see the description on page 15).

Contextual Data

Contextual data (also referred to as context) provides information about specific objects of a log record. Objects can be machines, users, or applications. Each object has many attributes that can describe it. Machines, for example, can be characterized by IP addresses, host names, autonomous systems, geographic locations, or owners.

Let’s take NetFlow records as an example. These records contain IP addresses to describe the machines involved in the communication. We wouldn’t know anything more about the machines from the flows themselves. However, we can use an asset context to learn about the role of the machines. With that extra information, we can make more meaningful statements about the flows—for example, which ports our mail servers are using.

Contextual data can be contained in various places, including asset databases, configuration management systems, directories, or special-purpose applications (such as HR systems). Windows Active Directory is an example of a directory that holds information about users and machines. Asset databases can be used to find out information about machines, including their locations, owners, hardware specifications, and more.

Contextual data can also be derived from log records; DHCP is a good example. A log record is generated when a machine (represented by a MAC address) is assigned an IP address. By looking through the DHCP logs, we can build a lookup table for machines and their IP addresses at any point in time. If we also have access to some kind of authentication information—VPN logs, for example—we can then argue on a user level, instead of on an IP level. In the end, users attack systems, not IPs.

Other types of contextual data include vulnerability scans. They can be cumbersome to deal with, as they are often larger, structured documents (often in XML) that contain a lot of information about numerous machines. The information has to be carefully extracted from these documents and put into the object model describing the various assets and applications. In the same category as vulnerability scans, WHOIS data is another type of contextual data that can be hard to parse.

Contextual data in the form of threat intelligence is becoming more common. Threat feeds can contain information around various malicious or suspicious objects: IP addresses, files (in the form of MD5 checksums), and URLs. In the case of IP addresses, we need a mechanism to expire older entries. Some attributes of an entity apply for the lifetime of the entity, while others are transient. For example, a machine often stays malicious for only a certain period of time.

Contextual data is handled separately from log records because it requires a different storage model. Mostly the data is stored in a key-value store to allow for quick lookups. For further discussion of quick lookups, see page 17.

How Much Data Do We Have in Total?

Just because everyone is talking about Hadoop doesn’t necessarily mean we need a big data solution to store our data. We can store multiple terabytes in a relational database, such as MySQL. Even if we need multiple machines to deal with the data and load, often sharding can help.

How Fast Does the Data Need to Be Ready?

In some cases, we need results immediately. If we drive an interactive application, data-retrieval rates often need to be completed at subsecond speed. In other cases, it is OK to have the result available the next day. Determining how fast the data needs to be ready can make a huge difference in how it needs to be stored.

How Much Data Do We Query, and How Often?

If we need to run all of our queries over all of our data, that is a completely different use-case from querying a small set of data every now and then. In the former case, we will likely need some kind of caching and/or aggregate layer that stores precomputed data so that we don’t have to query all the data at all times. An example is a query for a summary of the number of records seen per user per hour. We would compute those aggregates every hour and store them. Later, when we want to know the number of records that each user looked at last week, we can just query the aggregates, which will be much faster.

Where Is the Data and Where Does It Come From?

Data originates from many places. Some data sources write logs to files, others can forward data to a network destination (for example, through syslog), and some store records in a database. In some cases, we do not want to move the data if it is already stored in some kind of database and it supports our access use-case; this concept is sometimes called a federated data store.

What Do You Want with the Data and How Do You Access It?

While we won’t be able to enumerate every single use case for querying data, we can organize the access paradigms into five groups:

Search

Data is accessed through full-text search. The user looks for arbitrary text in the data. Often Boolean operators are used to structure more advanced searches.

Analytics

These queries require slicing and dicing the data in various ways, such as summing columns (for example, for sales prices). There are three subgroups:

Record-based analytics

These use cases entail all of the traditional questions we would ask a relational database. Business intelligence questions, for example, are great use cases for this type of analytics.

Relationships

These queries deal with complex objects and their relationships. Instead of looking at the data on a record-by-record (or row) basis, we take an object-centric view, where objects are anything from machines to users to applications. For example, when looking at machine communications, we might want to ask what machines have been communicating with machines that our desktop computer has accessed. How many bytes were transferred, and how long did each communication last? These are queries that require joining log records to come up with the answers to these types of questions.

Data mining

This type of query is about running jobs (algorithms) against a large set of our data. Unlike in the case of simple statistics, where we might count or do simple math, analytics or data-mining algorithms that cluster, score, or classify data fall into this category. We don’t want to pull all the data back to one node for processing/analytics; instead, we want to push the code down to the individual nodes to compute results. Many hard problems are related to data locality, and communication between nodes to exchange state, for example, that need to be considered for this use case (but essentially, this is what a distributed processing framework is for).

Raw data access

Often we need to be able to go back to the raw data records to answer more questions with data that is part of the raw record but was not captured in parsed data.

These access use cases are focused around data at rest—data we have already collected. The next two are use cases in the real-time scenario.

Real-time statistics

The raw data is not always what we need or want. Driving dashboards, for example, require metrics or statistics. In the simplest cases of real-time scenarios, we count things—for example, the number of events we have ingested, the number of bytes that have been transferred, or the number of machines that have been seen. Instead of calculating those metrics every time a dashboard is loaded—which would require scanning a lot of the data repeatedly—we can calculate those metrics at the time of collection and store them so they are readily available. Some people have suggested calling this a data river.

A commonly found use case in computer security is scoring of entities. Running models to identify how suspicious or malicious a user is, for example, can be done in real time at data ingestion.

Real-time correlation

Real-time correlation, rules, and alerting are all synonymous. Correlation engines are often referred to as complex event processing (CEP) engines; there are many ways of implementing them. One use case for CEP engines is to find a known pattern based on the definition of hard-coded rules; these systems need a notion of state to remember what they have already seen. Trying to run these engines in distributed environments gets interesting, especially when you consider how state is shared among nodes.

Using Parsers

Before we dive into details of how to store data, we need to discuss parsers. Most analysis requires parsed, or structured, data. We therefore need a way to transform our raw records into structured data. Fields (such as port numbers or IP addresses) inside a log record are often self-evident. At times, it’s important to figure out which field is the source address and which one is the destination. In some cases, however, identifying fields in a log record is impossible without additional knowledge. For example, let’s assume that a log record contains a number, with no key to identify it. This number could be anything: the number of packets transmitted, number of bytes transmitted, or number of failed attempts. We need additional knowledge to make sense of this number. This is where a parser adds value. This is also why it is hard and resource-intensive to write parsers. We have to gather documentation for the data source to learn about the format and correctly identify the fields. Most often, parsers are defined as regular expressions, which, if poorly written or under heavy load, can place a significant burden on the parsing system.

All kinds of off-the-shelf products claim that they don’t need parsers. But the example just outlined shows that at some point, a parser is needed (unless the data already comes in some kind of a structured form).

We need to keep two more things in mind. First, parsing doesn’t mean that the entire log record has to be parsed. Depending on the use case, it is enough to parse only some of the fields, such as the usernames, IP addresses, or ports. Second, when parsing data from different data sources, a common field dictionary needs to be used; this is also referred to as an ontology (which is a little more than just a field dictionary). All the field dictionary does is standardize the names across data sources. An IP address can be known by many names, such as: sourceAddress, sourceIP, srcIP, and src_ip. Imagine, for example, a setup where parsers use all of these names in the same system. How would you write a query that looked for addresses across all these fields? You would end up writing this crazy chain of ORed-together terms; that’s just ugly.

One last thing about parsing: we have three approaches to parsing data:

Collection-time parsing

In collection-time parsing, the data is parsed as soon as it is collected. All processing is then done on parsed, or structured, data—enabling all kinds of analytical use-cases. The disadvantage is that parsers have to be available up front, and if there is a mistake or an omission in the parsers, that data won’t be available.

Batch parsing

In batch parsing, the data is first stored in raw form. A batch process is then used to parse the data at regular intervals. This could be done once a day or once a minute, depending on the requirements. Batch parsing is similar to collection-time parsing in that it requires parsers up front, and after the data is parsed, it is often hard to change. However, batch parsing has the potential to allow for reparsing and updating the already-parsed records. We need to watch out for a few things, though—for example, computations that were made over “older” versions of the parsed data. Say we didn’t parse the username field before. All of our statistics related to users wouldn’t take these records into account. But now that we are parsing this field, those statistics should be taken into account as well. If we haven’t planned for a way to update the old stats in our application, those numbers will now be inconsistent.

Process-time parsing

Process-time parsing collects data in its raw form. If analytical questions are involved, the data is then parsed at processing time. This can be quite inefficient if large amounts of data are queried. The advantage of this approach is that the parsers can be changed at any point in time. They can be updated and augmented, making parsing really flexible. It also is not necessary to know the parsers up front. The biggest disadvantage here is that it is not possible to do any ingest-time statistics or analytics.

Overall, keep in mind that the topic of parsing has many more facets we don’t discuss here. Normalization may be needed if numerous data sources call the same action by different names (for example, “block,” “deny,” and “denied” are all names found in firewall logs for communications that are blocked). Another related topic is value normalization, used to normalize different scales. (For example, one data source might use a high, medium, or low rating, while another uses a scale from 1 to 10.)

Storing Log Data

To discuss how to store data, let’s revisit the access use cases we covered earlier (see What Do You Want with the Data and How Do You Access It?), and for each of them, discuss how to approach data storage.

Storing Context

We mentioned earlier that context, or contextual data, is slightly special. One option is to store context in a graph database by attaching the context as properties of individual objects. What if we want or need to use one of the other data stores? How do we leverage context for search, analytics, and data mining (distributed processing)? For example, we might want to search for all of the records that involve web servers. The data records themselves contain only IP addresses and no machine roles. The context, however, consists of a mapping from IP addresses to machine roles. Or we might want to compute some kind of statistics for web servers.

We can take two approaches to incorporate context for those cases:

Enrich at collection time

The first option is to augment every record at ingestion time. When the data is collected, we add the extra information to each record. This puts more load on the input processing and definitely consumes more storage on the back end, because every record now contains extra columns with the context. The benefit is that we can easily run any analytics/search right against the main data without any lookups. The caveat is that enrichment at collection time works only for context that does not change over time.

Enrich in batch: Instead of doing all of the enrichment in real time when data is collected, we can also run batch jobs over all of the data to enrich it anytime after it’s collected.

Join at processing time

The other option, if the overhead of enrichment is too high, is to join the data at processing time. Let’s take our example from earlier. We would first use the context store to find all the IP addresses of web servers. We would then take that list to query against the main data store to find all the records that involve those IP addresses. Depending on the size of the list of IP addresses (how many web servers there are), this can get pretty expensive in terms of processing time. In a relational data store that supports joins, we can normalize the schema and store the context in a separate table that is then joined against the main data table whenever needed.

Either or both of the preceding approaches can be right, depending on the situation. Most likely, you will end up with a hybrid approach, whereby some of the data should be enriched at collection time, some on a regular basis in batch, and some that is looked up in real time. The decision depends on how often we use the information and how expensive the query becomes if we do a real-time lookup.

Ideally, we implement a three-tier system:

1. Real-time lookup table: Lookups are often stored in key-value stores, which are really fast at finding associations for a key. Keep in mind that the reverse—looking up a key for a given value—is not easily possible. However, a method called an inverse index, which some key-value stores support out of the box, will facilitate this task. In other cases, you will have to add the inverse index (value→key) manually. In a relational database, you can store the lookups in a separate table. In addition, you might want to index the columns for which you issue a lot of lookups. Also, keep in mind that some lookups are valid at only certain times, so keep a time range with the data that defines the validity of the lookup. For example, a DHCP lease is valid for only a specific time period and might change afterward.

2. In-memory cache: Some lookups we have to repeat over and over again, and hitting disks to answer these queries is inefficient. Figure out which lookups you do a lot and cache those values in memory. This cache can be an explicit caching layer (something like memcache), or could be part of whatever key-value store we use to store the lookups.

3. Enrich data: The third tier is to enrich the data itself. Most likely there will be some data fields for which we have to do this to get decent query times across analytical and search operations. Ideally, we’d be able to instrument our applications to see what kinds of fields we need a lot and then enrich the data store with that information—an auto-adopting system.

Traditional Data Lake

Whatever data possible is stored in its raw form on HDFS. From there, it is picked up and forwarded into the SIEM, applying some filters to reduce the amount of data collected via the SIEM (see Figure 1-2).

The one main benefit of this architecture is that we can significantly reduce the effort of getting access to data for security monitoring tools. Without such a central setup, each new security monitoring tool needs to be fed a copy of the original data, which results in getting other teams involved to make configuration changes to products, making changes to production infrastructure that are risky, and having some data sources that might not support copying their data to multiple destinations. In the traditional data lake setup, data access can be handled in one place. However, this architecture has a few disadvantages:

• We need transport agents that can read the data at its origin and store it in HDFS. A tool called Apache Flume is a good option.

• Each product that wants to access the data lake (the raw data) needs a way to read the data from HDFS.

• Parsing has to be done by each product independently, thereby duplicating work across all of the products.

• When picking up the data and forwarding it to the SIEM (or any other product), the SIEM needs to understand the data format (syntax). However, most SIEM connectors (and products) are built such that a specific connector (say for Check Point Firewall) assumes a specific transport to be present (OPSEC in our example) and then expects a certain data format. In this scenario, the transport would not be correct.

For other data sources that we cannot store in HDFS, we have to get the SIEM connectors to directly read the data from the source (or forward the data there). In Figure 1-2 we show an arrow with a dotted line, where it might be possible to send a copy of the data into the raw data store as well.

As you can see, the traditional data lake setup doesn’t have many benefits. Hardly any products can read from the data lake (that is, HDFS), and it is hard to get the SIEMs to read from it too. Therefore, a different architecture is often chosen, whereby data is preprocessed before being collected.

Preprocessed Data

The preprocessed data architecture collects data in a structured or semistructured data store, before it is forwarded to the SIEM. Often this precollection is done either in a log management, or some other kind, of data warehouse, such as a columnar database. The data store is used to either summarize the data and forward summarized information to the SIEM, or to forward a filtered data stream in order to not overload the SIEM (see Figure 1-3).

The reasons for using a preprocessed data setup include the following:

• Reduces the load on the SIEM by forwarding only partial information, or forwarding presummarized information.

• Collects the data in a standard data store that can be used for other purposes; often accessible via SQL.

• Stores data in an HDFS cluster for use with other big data tools.

• Leverages cheaper data storage to collect data for archiving purposes.

• Frequently chosen if there is already a data warehouse, or a relational data store available for reuse.

As with a traditional data lake, some of the challenges with using the preprocessed data setup include the following:

• You will need a way to parse the data before collection. Often this means that the SIEM’s connectors are not usable.

• The SIEM needs to understand the data forwarded from the structured store. This can be a big issue, as discussed previously. If the SIEM supports a common log format, such as the Common Event Format (CEF), we can format the data in that format and send it to the SIEM.

Split Collection

The split collection architecture works only if the SIEM connector supports forwarding textual data to a text-based data receiver in parallel to sending the data to the SIEM. You would configure the SIEM connector to send the data to both the SIEM and to a process, such as Flume, logstash, or rsyslog, that can write data to HDFS, and then store the data in HDFS as flat files. Make sure to partition the data into directories to allow for easy data management. A directory per day and a file per hour is a good start until the files get too big, and then you might want to have directories per hour and a file per hour (see Figure 1-4).

Some of the challenges with using split collection include the following:

• The SIEM connector needs to have two capabilities: forwarding textual information and copying data to multiple destinations.

• If raw data is forwarded to HDFS, we need a place to parse the data. We can do this in a batch process (MapReduce job) over HDFS. Alternatively, some SIEM connectors are capable of forwarding data in a standardized way, such as in CEF format. (Having all of the data stored in a standard format in HDFS makes it easy to parse later.)

• If you are running advanced analytics outside the SIEM, you will have to consider how the newly discovered insight gets integrated back into the SIEM workflow.

Federated Data Access

It would be great if we could store all of our data in the data lake, whether it be security-related data, network metrics (the SNMP input in the diagram), or even HR data. Unlike in our first scenario (the traditional data lake), the data collected is not in raw form anymore. Instead, we’re collecting processed data (see Figure 1-5).

To enable access to the data, a “dispatcher” is needed to orchestrate the data access. As shown in Figure 1-5, not all data is forwarded to the lake. Some data is kept in its original store, and is accessed remotely by the dispatcher when needed.

Some of the challenges with using federated data access include the following:

• There is no off-the-shelf dispatcher available; you will need to implement this capability yourself. It needs to support both batch data access (probably through SQL), but also a real-time streaming capability to forward data to any kind of real-time system, such as your SIEM.

• Security products (such as behavior analytics tools, visualization tools, and search interfaces) need to be rewritten to leverage the dispatcher.

• Accessing data in a federated way (for example, HR data) might not be possible or may be hard to implement (for example, schemas need to be understood or systems need to allow third-party access).

• Controlling access to and protecting the data store becomes a central security problem, and any data-lake project will need to address these issues.

Despite all of the challenges with a federated data access setup, the benefits of such an architecture are quite interesting:

• Data is collected only once.

• Data from critical systems, such as an HR system, can be left in its original data store.

• The data lake can be leveraged by not only the security teams, but also any other function in the company that needs access to the same data.

A fifth setup consists of first collecting the data in a SIEM and then extracting it to feed it into the security data lake. This setup is somewhat against the principle of the data lake, in that it first collects the data in a big data setup and then gives third-party tools (among them the SIEM) access. In addition, most SIEMs do not have a good way to get data out of their data store.