kafka-go: the kafka reader got an unknown error reading partition
First issue
we’re often getting a reader error ~500k/day.
the kafka reader got an unknown error reading partition 9 of SOME_TOPIC at offset 3: read tcp IP_ADDRESS:46406->IP_ADDRESS:9093: i/o timeout
Kafka-go 0.4.16
Kafka 2.5.0
Second issue
The errors happened when kafka reader is committing the message after it has been processed successfully, the message was re-consumed by another replica
“msg”: “debezium.Consumer: failed to commit message: write tcp IP_ADDRESS:49610->IP_ADDRESS:9093: use of closed network connection” Receiving Successfully handled message and a while afterward getting failed to commit message
About this issue
- Original URL
- State: open
- Created 5 years ago
- Comments: 17 (7 by maintainers)
Commits related to this issue
- #726 remove reader safetyTimeout — committed to moogacs/kafka-go by moogacs 4 years ago
- #726 remove reader safetyTimeout — committed to moogacs/kafka-go by moogacs 4 years ago
- #726 remove reader safetyTimeout — committed to moogacs/kafka-go by moogacs 4 years ago
- #726 make kafka reader message ReadBatchTimeout configurable (#989) * #726 remove reader safetyTimeout * add configuration * update docs and add default * update comment * add read deadl... — committed to segmentio/kafka-go by moogacs 4 years ago
#989 has been merged, let me know if you are still experiencing the issue on the latest version of kafka-go!
@mostafa, the issue has been detected and indeed allowing to configure the reads timeout will fix is so the user of the lib will be able to adjust the read timeouts based on their needs.
atm it’s hard coded to 10 sec. and increasing that will resolve it.
So I am waiting in approving my PR and merging it
And not sure the meaning of code below
it overwrite the deadline which been set before and cannot be configured with MaxWait in Reader configure. Can we remove the code above? Refer to commented code below
@achille-roussel i think can help in peroritizing https://github.com/segmentio/kafka-go/pull/989
@achille-roussel I am in the same team/company as @nsomarouthu, working together on the 2 issues. For the 1st issue, we have many
the kafka reader got an unknown error reading partitionas well as numerousno messages received from kafka within the allocated time for partitionlogged. I believe this is because our applications setReaderConfig.MaxWaithttps://github.com/segmentio/kafka-go/blob/master/reader.go#L395 to 100ms. There are some messages that we want the application to fetch and handle as soon as possible. Do you see any concern with setting a shortReaderConfig.MaxWait? DoesReaderConfig.MaxWaitduration impact the expect message consumption rate? (We only need a shortReaderConfig.MaxWaitfor some of our apps, not all of our apps).For the 2nd issue, we have a few error messages about 5 seconds after finished handling and trying to commit the Kafka message with error such as
write tcp 100.96.92.51:52652->100.67.54.193:9093: use of closed network connection. I believe this is related toConsumerGroupConfig.Timeout(default to 5s) https://github.com/segmentio/kafka-go/blob/master/consumergroup.go#L161 Is there a way to set a different (longer) value when creating aNewReaderwith aReaderConfig? It does not look like it is configurable here: https://github.com/segmentio/kafka-go/blob/master/reader.go#L704