aiokafka: Skip and proceed in the case of CorruptRecordException
I get the following error while consuming and then consumption stops
records = res_or_error.getall(max_records)
File "/home/foo/env/lib/python3.6/site-packages/aiokafka/consumer/fetcher.py", line 100, in getall
for msg in chain([first_msg], next_batch_iter):
File "/home/foo/env/lib/python3.6/site-packages/aiokafka/consumer/fetcher.py", line 850, in _unpack_records
raise Errors.CorruptRecordException("Invalid CRC")
kafka.errors.InvalidMessageError: [Error 2] InvalidMessageError: Invalid CRC
I wonder if there should be a flag to inform the consumer to keep going forward in the case of a corrupted message instead of getting stuck.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 24 (23 by maintainers)
You are right I think it was a disk issue. We made some changes with our kafka setup and the hardware and this was fixed.