Kafka Connect

Chapter 8. Configuring Kafka Connect

Throughout this book, we mention configuration settings that can be used to customize Kafka Connect. All of these settings are documented on the Apache Kafka website, where you can find their names, types, default values, and descriptions.

In this chapter, we go over all the configuration settings and provide additional context to help you identify the key settings to be aware of. We also show how some of these configurations can be combined to achieve the behaviors required for more advanced use cases.

We first look at the runtime, from its most basic configurations that are necessary to start it to more specific settings that affect the behavior of connectors. We then look at configuring connectors, again starting from the basics and then covering more advanced concepts like client overrides and error handling. Finally, we look at all the configurations available to secure Kafka Connect clusters for production use cases.

In this chapter, we refer to both “runtime configuration” and “worker configuration.” Since a worker is really just a single instance of the runtime, these terms are somewhat interchangeable. However, we aim to use “runtime configuration” when talking generically about how configuration affects the Kafka Connect runtime and “worker configuration” when talking about the specific configuration that you use to start a worker.

Note

Since we recommend using distributed mode for production clusters, we don’t specifically call out ...