kafka-go: Multiple consumer group offset problems with 0.4.36

Describe the bug

Several consumer group issues with ReadMessage and FetchMessage when using lib version 0.4.36:

  1. Offsets are not committed at all when CommitMessages is called manually after a successful FetchMessage
  2. Offsets are partially committed when using ReadInterval + CommitInterval set to 0
  3. Downgrading to 0.4.35 fixes both of the issues

Kafka Version

2.4.1.1

To Reproduce

groupID := "foo"

readerConfig := kafka.ReaderConfig{
	Brokers: k.Options.Brokers,
	GroupID: groupID,
	Topic:   topic,
	MaxWait: k.Options.ReaderMaxWait,
	Dialer:  k.Dialer,
	CommitInterval: 0,
}

reader := kafka.NewReader(readerConfig) 

// Try to read some messages
m, err := reader.ReadMessage(context.Background())
if err != nil {
    // err
}

// Close reader, create new one w/ same GroupID
m, err := reader.ReadMessage(context.Background())
if err != nil {
    // err
}

// A dupe read will occur

// If you view kafdrop or use another utility to view lag - the consumer group will likely have lag as well.

// When using FetchMessage and calling CommitMessages - offsets do not appear to be committed at all.

Thank you!

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 8
  • Comments: 16 (3 by maintainers)

Most upvoted comments

https://github.com/segmentio/kafka-go/pull/1022 Contains some potential fixes for some of the issues 0.4.36 but I’m still not able to reproduce any issues with committing offsets in MSK, Strimzi or running Kafka locally. Some of the issues seem related to interaction with other members of consumer groups.

Is anyone using a mixture of clients within a single consumer group?

As always, logs are very helpful for debugging.

Thanks!

I am facing partially the same issue and it caused a problem on our side during this week due to duplicates that were generated in large quantity

offsets are committed partially

If it helps I started receiving multiple warnings also after this upgrade

I am suspecting this PR https://github.com/segmentio/kafka-go/pull/947

I agree on this aspect too as I see memberID being introduced in this PR but not entirely sure of the change

 Failed to join group identifier-ingester: [79] Member ID Required: the group member needs to have a valid member id before actually entering a consumer group