confluent-kafka-go: Consumer.Close() never returns with multiple consumers in the same process

I’m running 2 consumers on the same topic and the same consumer group, using Poll() to consume messages. After having consumed messages with both of them, calling Close() on any of the consumers blocks indefinitely, reproducible 100% of the time. Debug logs show things being stuck at “waiting for rebalance_cb”

This is the gist with the go code to reproduce and all relevant information: https://gist.github.com/dtheodor/26821f951502aab2e325f860b459cbfc

This could be the same issue as https://github.com/confluentinc/confluent-kafka-go/issues/65 but you are talking about using the Events channel there, which I am not.

Calling Unassign() right before calling Close() allows Close() to return successfully, but having to call Unassign looks bogus since I never have to use Assign or Unassign in any other place, this is managed by the lib for me.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 5
  • Comments: 35 (23 by maintainers)

Commits related to this issue

Most upvoted comments

This issue is fixed in librdkafka adminapi branch which will be merged to master during this week and released in june.

The workaround for now is to drain the consumer queue by calling Poll() until nil is returned, which means all outstanding callbacks (and messages) have been processed.

My current solution is to use ver 0.11.3.

Calling Unsubscribe() (or actually Unassign(), which is called when the assignment is revoked) will purge all messages the in the pre-fetch queue.