Learning PySpark

What is the Spark Streaming application data flow?

The following figure provides the data flow between the Spark driver, workers, streaming sources and targets:

It all starts with the Spark Streaming Context, represented by ssc.start() in the preceding figure:

1. When the Spark Streaming Context starts, the driver will execute a long-running task on the executors (that is, the Spark workers).
2. The Receiver on the executors (Executor 1 in this diagram) receives a data stream from the Streaming Sources. With the incoming data stream, the receiver divides the stream into blocks and keeps these blocks in memory.
3. These blocks are also replicated to another executor ...