Implementing a Smart Data Platform

Chapter 6. SmartDP Reference Architecture

In terms of architecture, SmartDP should be able to support data management, data science, and data engineering, and enable data engineering teams to effectively collaborate with one another (see Figure 6-1).

Functionally, SmartDP is divided into five layers, as shown in Figure 6-1—the data layer, data access layer, infrastructure layer, data application layer, and operation management layer. Let’s look at each layer in detail.

Figure 6-1. SmartDP reference architecture (figure courtesy of Wenfeng Xiao)

Data Layer

The data layer may be divided into self-owned data and third-party data, according to sources.

Self-Owned Data

Self-owned data refers to data that is owned by an enterprise and that can be completely controlled and managed by the enterprise, which is stored in the enterprise or any external platform of a third party.

The self-owned data stored in an enterprise generally includes:

• Data from mobile smart devices, including user interaction data of native smart phone applications (including Android, iOS, WinPhone) and of vertical application frameworks (including Cocos2D, Unity3D, APICloud, Cordova).

• Data from web pages, including user click events of PC or mobile web pages (such as JavaScript) and web page information crawled from web pages

• Data from IoT devices, including beacon devices (such as iBeacon and Eddystone) ...