ruby-kafka: Consumer does not consume messages before all leader brokers receive a message

Experiencing strange bug/behavior from consumer (group API) with the following setup:

1 consumer subscribed to a topic. 3 brokers, 10 partitions, replication factor of 2.

When message is appended, the consumer does not consume it immediately. Only when all 3 brokers get at least one message does the consumer start consuming. (min_bytes is set to 1)

Further testing with 2 partition only, revealed the following: if there is 1 leader for both partitions, messages are consumed immediately. If partitions have different leaders, messages are not consumed until both leaders get the messages.

Consumer example:

...
consumer = kafka.consumer(group_id: group_id)

trap('INT') { consumer.stop } # ctrl-c
trap('TERM') { consumer.stop } # kill PID

consumer.subscribe(
  topic,
  start_from_beginning: false
)

consumer.each_batch(min_bytes: 1, max_wait_time: 5) do |batch|
  batch.messages.each do |message|
     puts message.value
  end
end

Expected behavior: Consumer should start consuming immediately after message is appended.

Note: Seems like an issue in the gem, because when using a cli consumer (kafka-console-consumer.sh), messages are consumed immediately after being appended.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 30 (17 by maintainers)

Most upvoted comments

@dasch We tried with 0.3.18.beta2, but it does not solve the problem.