The following diagram represents the Kafka Connect architecture:
The Kafka cluster is made of Kafka brokers: three brokers, as shown in the diagram. Sources can be of any type, for example, databases, NoSQL, Twitter, and so on. In between the source and Kafka cluster, there is a Kafka Connect cluster, which is made up of workers. The working of Kafka Connect comprises the following steps:
- Workers, based on configuration, pull data from sources
- After getting data, the connector pushes data to the Kafka cluster
- If data needs to be transformed, filtered, joined, or aggregated using stream applications such as Spark, ...
