confluent-kafka-dotnet: Consumer error Local_MaxPollExceeded is Fatal for consumer
Description
When my code got stuck (dumping, debugging, lack of system resources, bad code…), and as a result of it I do not call Consume method on Consumer longer than max.poll.interval.ms Consumer throws exception with error
Code=Local_MaxPollExceeded
IsBrokerError=false
IsError=true
IsFatal=false
IsLocalError=true
Consumer does not recover, and I need to restart my app (I suppose recreating consumer would also work). From my point of view, shouldn’t this error be Fatal? If not why Consumer does not recover?
How to reproduce
Consume some message from topic, then break app using Thread.Sleep() for longer interval than max.poll.interval.ms
Checklist
Please provide the following information:
- A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
- Confluent.Kafka nuget version. 1.3.0
- Apache Kafka version. kafka_2.12-2.2.0
- Operating system. - windows/linux
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 18 (8 by maintainers)
Commits related to this issue
- WIP: max.poll.interval.ms should only be enforced when using subscribe() For https://github.com/confluentinc/confluent-kafka-dotnet/issues/1220 — committed to confluentinc/librdkafka by edenhill 4 years ago
- max.poll.interval.ms should only be enforced when using subscribe() For https://github.com/confluentinc/confluent-kafka-dotnet/issues/1220 — committed to confluentinc/librdkafka by edenhill 4 years ago
- max.poll.interval.ms should only be enforced when using subscribe() For https://github.com/confluentinc/confluent-kafka-dotnet/issues/1220 — committed to confluentinc/librdkafka by edenhill 4 years ago
Ah, I think I know what the problem is, you are not using Subscribe(), but Assign()ing partitions directly. The max.poll.interval.ms should not be enforced unless there is an active group subscription, this is a bug in librdkafka that we’ll fix for v1.4.0. The workaround is to set max.poll.interval.ms to its maximum value.
One thing to note - we do not use consumer groups and assign the start offset manually.