Errata

Following are the exact lines written on delivery.timeout.ms in chapter 3(kafka producers), 2nd edition, kafka, the definitive guide.

delivery.timeout.ms >=linger.ms +retry.backoff.ms +request.timeout.ms
If producer exceeds delivery.timeout.ms while the record batch was still waiting to be sent, the callback will be called with a timeout exception.

However, in reality, prod application is showing a different behavior.
Here is an exact details from my prod config and the details of how it behaves:
delivery.timeout.ms = 120000( = 2mins=default)
linger.ms=0
request.timeout.ms=3600000 (60 mins)
I am observing a timeout exception :
org.apache.kafka.common.errors.TimeoutException: Expiring 14 records for topic1 3600008 ms has passed since batch creation.
As you can see here the producer is still waiting for 60 mins(= request.timeout.ms ) regardless of the value of delivery.timeout.ms
This contradicts what is explained in the book.
The book says the producer will throw TimeoutException as soon as the delivery.timeout.ms seconds has elapsed.
In this case, delivery.timeout.ms=2 mins ,so i should have got an exception in 2 mins instead of getting it after 60 mins.
I think the thread that's trying to submit the record does not care about delivery.timeout.ms ; rather it's looking for request.timeout.ms value.
Does the TimeoutException gets triggered only when producers try to add new records to a previous batch that is still waiting to be sent ?
Looks like there is no way for the client/producer to wait for the current batch to be sent successfully to brokers before sending the next batch as the send() method is returning without making sure the messages are sent successfully to broker.

Can you explain how it works ?

Below is the definition of synchronous send from your book:
Technically, Kafka producer is always asynchronous—we send a message and the send() method returns a Future object. However, we use get() to wait on the Future and see if the send() was successful or not before sending the next record.

However, even if the get() method return is successful , the producer still throws "org.apache.kafka.common.errors.TimeoutException: Expiring 14 records for topic1 3600008 ms has passed since batch creation"
Below are the configuration:
delivery.timeout.ms = 120000( = 2mins)
linger.ms=0
request.timeout.ms=3600000 (60 mins)

question:
what is the best way to wait for the producer to wait for the batch to be sent successfully before moving onto the next batch ?
My understanding was future.get() returns success only when the broker receives it but it seems like my assumption is not true.
can you explain more on what is going on here ?

Sir,

I had gone through Kafka - The Definitive Guide (2nd Ed) and found it to be a great book. However, I do have a suggestion for the authors. Can you please highlight it to them?

The codes for the entire book is in Java. However, now days, even Python is being used in entire Data Science. Keeping this in mind, can the authors come out with an edition of this great book which will have Python codes as well, along with Java. Also, as of now, they can update the Github repo of this book with compliant Python codes.

I hope my suggestion is good enough keeping in mind about the future.