Stream Processing with Apache Spark

Part II. Structured Streaming

In this part, we examine Structured Streaming.

We begin our journey by exploring a practical example that should help you build your intuition for the model. From there, we examine the API and get into the details of the following aspects of stream processing:

• Consuming data using sources

• Building data-processing logic using the rich Streaming Dataframe/Dataset API

• Understanding and working with event time

• Dealing with state in streaming applications

• Learning about arbitrary stateful transformations

• Writing the results to other systems using sinks

Before closing, we provide an overview of the operational aspects of Structured Streaming.

Finally, we explore the current developments in this exciting new streaming API and provide insights into experimental areas like machine learning applications and near-real-time data processing with continuous streaming.