Intel’s internal IoT platform for real-time enterprise analytics
A single, multitenant platform built with open source technologies, based on an understanding of basic common needs.
Vippen. (source: Christian Theede Christiansen on Flickr)
Recent years have seen a significant evolution of the Internet of Things (IoT). It has become increasingly easy to connect devices to the Internet and send sensorial data to the public cloud. However, it’s quite evident that the adoption of IoT platforms and stream analytics within the enterprise is lagging and less prevalent. This could be associated with lack of skills required to deploy a platform and the difficulty to demonstrate value of such platforms through real-life use cases.
Most enterprises can benefit from an internal IoT implementation, and most are already practicing analytics at different levels from descriptive to predictive/prescriptive, and stream analytics should be part of their analytics stack. Stream analytics can be applied effectively to enable closed-loop systems that promote proactivity and operational excellence.
Learn faster. Dig deeper. See farther.
We at Intel identified a few potential projects and use cases of IoT in different domains such as:
- Manufacturing and related activities – IoT enables Smarter Factories
- Smart Buildings: IoT can help fulfill the potential and realize the value
- Asset management: RFID-enabled inventory management
- Preventive maintenance: applicable to just about any tool / machine / device
In addition, we found that some operational activities may be “translated” into IoT stream analytics scenarios to allow a higher level of proactivity and a shift from manual monitoring and firefighting to higher value work.
Previous experience with IoT projects has shown that the natural tendency is to treat different IoT implementations as separate and unique. The common assumption is that the different needs cannot be addressed with one platform. This might be a misperception. Indeed, different use cases have different requirements, however most IoT projects have a basic set of common needs that may be addressed with one reusable platform.
Hierarchy of needs
Based on our learnings from various use cases with different customers, we have identified a “hierarchy of needs” for IoT solutions that can help to highlight the commonalities of apparently unrelated and very different IoT use cases.
We identified five main layers or needs in this hierarchy (Figure 1). In most cases, they appear in this order and priority:
- Flow Data: At the basis level, there is the very basic need to flow data from the edge device into the target database in the cloud as a means to collect the data.
- Visualize: The next need is to be able to visualize the data in real time, track the data using charts and dashboards, and react to irregular events. This helps build trust in the system’s reliability. However, it’s quite clear that this approach by itself is not scalable; people can’t monitor hundreds of metrics or charts 24×7, which leads to the next need in the hierarchy.
- Monitor: We need to have the ability to put automated monitors on streaming data. Users should be able to easily define rules or conditions that, if met, will trigger an automatic actuation or alert in near-real time. This is a powerful capability that leverages humans’ domain expertise and intuition.
- Auto-Detect: In many cases the rules of automated monitoring are not known to the user or are very hard to define. This is where predictive analytics can play a role. With a combination of machine learning algorithms and statistical models, we can let the machine automatically detect changes or anomalies in the data and trigger action.
- Build & customize: Customize and build on top of the generic platform to add use case-specific functionalities.
It should be quite evident that the first four layers are fairly generic and could be the basis for a reusable IoT platform that can meet a significant portion of use cases out-of-the-box. By identifying the set of characteristics and needs that are common to many IoT scenarios, we were able to deploy an internal platform to be used in different use cases and business domains.
Intel’s internal IoT platform
IoT involves volumes and velocities of data that make it a viable big data use case. We found that our internal IoT platform can effectively use existing big data infrastructure and leverage previous investments. Our platform was based on several open source technologies, including Spark, Kafka, AKKA, and Hadoop. In addition, we used Docker and CoreOS to address DevOps considerations and allow portability of the platform (Figure 2).
To enable stream analytics at scale, we implemented an AKKA-based stream processing framework that allows us to analyze massive data streams from many devices simultaneously in near-real time. Spark Streaming was used to deploy self-service data monitors that allow users to define their own rules and actuation when a condition was met. These rules are monitored in near-real time on the stream. The analytics layer uses machine learning techniques and advanced statistical tests to provide meaningful insights to the users of the system. This layer also includes algorithms such as multi-sensor change detection, anomaly detection, and more.
The ability to implement a single multitenant IoT platform that can add value in different domains almost out of the box, combined with the ability to maximize the return on previous big data investments and existing platforms, may be a promising direction for enabling the anticipated IoT analytics revolution.
Moty Fania will speak in greater detail on this topic during his session Stream Analytics in the Enterprise: A Look at Intel’s Internal IoT Implementation at Strata + Hadoop World London 2016, May 31 to June 3, 2016.