aiokafka: Skip and proceed in the case of CorruptRecordException

I get the following error while consuming and then consumption stops

    records = res_or_error.getall(max_records)
  File "/home/foo/env/lib/python3.6/site-packages/aiokafka/consumer/fetcher.py", line 100, in getall
    for msg in chain([first_msg], next_batch_iter):
  File "/home/foo/env/lib/python3.6/site-packages/aiokafka/consumer/fetcher.py", line 850, in _unpack_records
    raise Errors.CorruptRecordException("Invalid CRC")
kafka.errors.InvalidMessageError: [Error 2] InvalidMessageError: Invalid CRC

I wonder if there should be a flag to inform the consumer to keep going forward in the case of a corrupted message instead of getting stuck.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 24 (23 by maintainers)

Most upvoted comments

You are right I think it was a disk issue. We made some changes with our kafka setup and the hardware and this was fixed.