Kafka Essentials LiveLessons: A Quick-Start for Building Effective Data Pipelines

Table of contents

1. Lesson 1: Kafka Background Concepts
   1. 1.1 Why Do I need a Message Broker?
   2. 1.2 Understanding Kafka Components
   3. 1.3 Running Kafka on the LHM Virtual Machine
   4. 1.4 Understand Basic Kafka Usage
2. Lesson 2: Viewing Kafka Operations
   1. 2.1 Exploring the KafkaEsque UI Features
   2. 2.2 Replay Basic Examples with KafkaEsque
3. Lesson 3: Streaming NOAA Weather Data with Kafka Python
   1. 3.1 Understand Components: Kafka, Python, NOAA-sdk
   2. 3.2 Construct a Python Producer using keys, synchronization, and serialization
   3. 3.3 Construct a Python Consumer using groups, partition rebalancing, commits and offsets
4. Lesson 4: Moving Kafka Topic Data to External Storage
   1. 4.1 Use a PySpark consumer to stream data to HDFS
   2. 4.2 Use a Python consumer to write data to MariaDB (MySQL)
   3. 4.3 Use a Python consumer to write data to HBASE
5. Lesson 5: Edge Image Streaming with Kafka Python
   1. 5.1 Understand detecting Edge 3D printing problems
   2. 5.2 Learn how to configure Kafka for Image streaming
   3. 5.3 Learn how to configure image capture, streaming, and real-time CNN analysis using Kafka producers and consumers