Kafka Connect

Chapter 11. Building Source and Sink Connectors

As mentioned in previous chapters, the Apache Kafka project does not include any connectors apart from MirrorMaker and the example FileStream connectors. Instead, the Kafka community and many vendors build connectors. There are now hundreds of them available for the most popular databases, data processing tools, storage tools, and protocols.

However, if there are no connectors for the systems you use, or if the existing connectors are not suitable for your use cases, you can build your own source and sink connectors. In this chapter, we describe the process of implementing a connector from scratch. We cover the classes in the Kafka Connect public API that you need to implement and explain the main steps, best practices, and decisions you need to make in order to build a connector that satisfies your requirements.

Source and sink connectors share a lot of the same building blocks, so many pieces of work and decision points are the same when building your own. Before diving into the specifics of each type of connector, we look at the concepts that apply to both types, such as how to build the connector and the common APIs.

Common Concepts and APIs

As explained in previous chapters, a connector consists of two parts: the connector and the tasks. The initial entry point is the connector, which then configures one or more tasks that actually do the work. These parts map to the Connector and Task classes that you need to implement in ...