kafka-python: CommitFailedError when there are very few messages to be read from topic

This CommitFailedError arises when there are very few messages to be read from topic … I’ve experiencing it since my early use of kafka-python (release 2.0.2) and it is still present with latest 2.2.3 . Since I need to process every message only once, this led me to disable autocommit and do instead a manual commit after each poll returning some messages, then process the read messages.

In the below example, “no record available” gets printed after each poll with no returned messages, so you can see poll() gets called every 5 seconds (or immediately after processing the last message).

02/05/2025; 18:49:02.94; [I] $GMT22 ; no record available
02/05/2025; 18:49:08.09; [I] $GMT22 ; no record available
02/05/2025; 18:49:13.23; [I] $GMT22 ; no record available
02/05/2025; 18:49:18.33; [I] $GMT22 ; no record available
02/05/2025; 18:49:23.53; [I] $GMT22 ; no record available
02/05/2025; 18:49:28.70; [I] $GMT22 ; no record available
02/05/2025; 18:49:34.29; [E] $GMT22 ; read records - error handling polled messages - CommitFailedError: Commit cannot be completed since the group has already
            rebalanced and assigned the partitions to another member.
            This means that the time between subsequent calls to poll()
            was longer than the configured max_poll_interval_ms, which
            typically implies that the poll loop is spending too much
            time message processing. You can address this either by
            increasing the rebalance timeout with max_poll_interval_ms,
            or by reducing the maximum size of batches returned in poll()
            with max_poll_records.

02/05/2025; 18:49:39.39; [I] $GMT22 ; no record available
02/05/2025; 18:49:44.50; [I] $GMT22 ; no record available
02/05/2025; 18:49:49.51; [W] $GMT22 ; read records - committed (1)
02/05/2025; 18:49:49.51; [W] $GMT22 ; read|oamds-mtonline|1|189230|b'no_check_1000200ffff33gfff'|10000
02/05/2025; 18:49:49.68; [I] $GMT22 ; nowait response for tag 10000
02/05/2025; 18:49:49.79; [I] $GMT22 ; no record available

max_poll_interval_ms is left to default value (300 seconds I think) and not even provided as a parameter to my Consumer.

Not sure if it has to do with the above behaviour, but at least two Kafka brokers administrators told me they do not see my client registered … even if my client is regularly polling and processing messages.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 30 (16 by maintainers)

Most upvoted comments

Fantastic ! I think you really got it … the python release (currently 3.11) provided by HPE is certainly CPython and the suggested trick with a 1ms time sleep after each poll seems to work. I’ll do some more testing next week since our test broker gets switched off over the weekend … but I could already see that there is no more commit error also with INFO logging and DEBUG logging shows hearbeats every 5 to 6 seconds.