kafka-go: Multiple consumer group offset problems with 0.4.36
Describe the bug
Several consumer group issues with ReadMessage
and FetchMessage
when using lib version 0.4.36
:
- Offsets are not committed at all when
CommitMessages
is called manually after a successfulFetchMessage
- Offsets are partially committed when using ReadInterval + CommitInterval set to 0
- Downgrading to
0.4.35
fixes both of the issues
Kafka Version
2.4.1.1
To Reproduce
groupID := "foo"
readerConfig := kafka.ReaderConfig{
Brokers: k.Options.Brokers,
GroupID: groupID,
Topic: topic,
MaxWait: k.Options.ReaderMaxWait,
Dialer: k.Dialer,
CommitInterval: 0,
}
reader := kafka.NewReader(readerConfig)
// Try to read some messages
m, err := reader.ReadMessage(context.Background())
if err != nil {
// err
}
// Close reader, create new one w/ same GroupID
m, err := reader.ReadMessage(context.Background())
if err != nil {
// err
}
// A dupe read will occur
// If you view kafdrop or use another utility to view lag - the consumer group will likely have lag as well.
// When using FetchMessage and calling CommitMessages - offsets do not appear to be committed at all.
Thank you!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 8
- Comments: 16 (3 by maintainers)
https://github.com/segmentio/kafka-go/pull/1022 Contains some potential fixes for some of the issues 0.4.36 but I’m still not able to reproduce any issues with committing offsets in MSK, Strimzi or running Kafka locally. Some of the issues seem related to interaction with other members of consumer groups.
Is anyone using a mixture of clients within a single consumer group?
As always, logs are very helpful for debugging.
Thanks!
I am facing partially the same issue and it caused a problem on our side during this week due to duplicates that were generated in large quantity
offsets are committed partially
If it helps I started receiving multiple warnings also after this upgrade
I agree on this aspect too as I see memberID being introduced in this PR but not entirely sure of the change