ruby-kafka: Consumer does not consume messages before all leader brokers receive a message
Experiencing strange bug/behavior from consumer (group API) with the following setup:
1 consumer subscribed to a topic. 3 brokers, 10 partitions, replication factor of 2.
When message is appended, the consumer does not consume it immediately. Only when all 3 brokers get at least one message does the consumer start consuming. (min_bytes
is set to 1)
Further testing with 2 partition only, revealed the following: if there is 1 leader for both partitions, messages are consumed immediately. If partitions have different leaders, messages are not consumed until both leaders get the messages.
Consumer example:
...
consumer = kafka.consumer(group_id: group_id)
trap('INT') { consumer.stop } # ctrl-c
trap('TERM') { consumer.stop } # kill PID
consumer.subscribe(
topic,
start_from_beginning: false
)
consumer.each_batch(min_bytes: 1, max_wait_time: 5) do |batch|
batch.messages.each do |message|
puts message.value
end
end
Expected behavior: Consumer should start consuming immediately after message is appended.
Note: Seems like an issue in the gem, because when using a cli consumer (kafka-console-consumer.sh), messages are consumed immediately after being appended.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 30 (17 by maintainers)
@dasch We tried with 0.3.18.beta2, but it does not solve the problem.