kafka-python: "import kafka" fails with "ModuleNotFoundError: No module named 'kafka.vendor.six.moves'" under Python 3.12

I have verified this on Linux and Windows, even though the output below is from Linux:

user1@CY1-WL-337:~/dev$ python3.12 -m venv venv
user1@CY1-WL-337:~/dev$ source venv/bin/activate
(venv) user1@CY1-WL-337:~/dev$ pip install kafka-python
Collecting kafka-python
  Using cached kafka_python-2.0.2-py2.py3-none-any.whl (246 kB)
Installing collected packages: kafka-python
Successfully installed kafka-python-2.0.2

(venv) user1@CY1-WL-337:~/dev$ python -c 'import kafka'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/__init__.py", line 23, in <module>
    from kafka.consumer import KafkaConsumer
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/__init__.py", line 3, in <module>
    from kafka.consumer.group import KafkaConsumer
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/group.py", line 13, in <module>
    from kafka.consumer.fetcher import Fetcher
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/fetcher.py", line 19, in <module>
    from kafka.record import MemoryRecords
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/__init__.py", line 1, in <module>
    from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/memory_records.py", line 27, in <module>
    from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/legacy_records.py", line 50, in <module>
    from kafka.codec import (
  File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/codec.py", line 9, in <module>
    from kafka.vendor.six.moves import range
ModuleNotFoundError: No module named 'kafka.vendor.six.moves'

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 37 (6 by maintainers)

Commits related to this issue

Most upvoted comments

This was patched in https://github.com/dpkp/kafka-python/pull/2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.

Please understand that I am doing everything within my means to get a release out to the public.

The following code fix works for me with the current public build of kafka-python:

import sys

if sys.version_info >= (3, 12, 0):
    import six
    sys.modules['kafka.vendor.six.moves'] = six.moves

......

from kafka import KafkaConsumer, KafkaProducer

......

Can we please reopen this bug because a critical bug like this should not be closed until the fix is released. I would even pin it after reopening it, just to remind everyone about it.

This is still not fixed, and have been open for a year now. Does everyone really install from the github repo? Is it so hard to do a release?

Anyway, commenting to show additional demand. Although demand for “be able to use the library” seems self-explanatory…

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime. Please understand that I am doing everything within my means to get a release out to the public.

@wbarnha Thank you William for your support here. Installing python -m pip install --break-system-packages git+https://github.com/dpkp/kafka-python.git still is not resolving the issue.

i can not downgrade python 3.11 due to a HIGH vulnerability with ecdsa pacakge. Any work around solution for this problem at the moment?

The current workaround solution would be to replace your kafka-python installation with my fork at kafka-python-ng for now.

This was patched in #2398 and will be resolved in the next release. I apologize for the delay. Hopefully running pip install git+https://github.com/dpkp/kafka-python.git could fulfill your needs in the meantime.

Please understand that I am doing everything within my means to get a release out to the public.

Is there an idea of when the next release will be out?

Hey guys, this solved it for me: 1- Uninstall kafka-python using: pip uninstall kafka-python 2- Then run: pip install --break-system-packages git+https://github.com/dpkp/kafka-python.git

Hi i am still having this issue. Is it resolved in the latest release.

here is my contect: Python 3.12.2 kafka-python 2.0.2

#import procedure: from kafka import KafkaProducer Error: ModuleNotFoundError Traceback (most recent call last) Cell In[7], line 1 ----> 1 from kafka import KafkaProducer

File ~\AppData\Lib\site-packages\kafka_init_.py:23 21 from kafka.admin import KafkaAdminClient 22 from kafka.client_async import KafkaClient —> 23 from kafka.consumer import KafkaConsumer 24 from kafka.consumer.subscription_state import ConsumerRebalanceListener 25 from kafka.producer import KafkaProducer

File ~\AppData\Lib\site-packages\kafka\consumer_init_.py:3 1 from future import absolute_import ----> 3 from kafka.consumer.group import KafkaConsumer 5 all = [ 6 ‘KafkaConsumer’ 7 ]

File ~\AppData\Lib\site-packages\kafka\consumer\group.py:13 10 from kafka.vendor import six 12 from kafka.client_async import KafkaClient, selectors —> 13 from kafka.consumer.fetcher import Fetcher 14 from kafka.consumer.subscription_state import SubscriptionState 15 from kafka.coordinator.consumer import ConsumerCoordinator

File ~\AppData\Lib\site-packages\kafka\consumer\fetcher.py:19 15 from kafka.protocol.fetch import FetchRequest 16 from kafka.protocol.offset import ( 17 OffsetRequest, OffsetResetStrategy, UNKNOWN_OFFSET 18 ) —> 19 from kafka.record import MemoryRecords 20 from kafka.serializer import Deserializer 21 from kafka.structs import TopicPartition, OffsetAndTimestamp

File ~\AppData\Lib\site-packages\kafka\record_init_.py:1 ----> 1 from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder 3 all = [“MemoryRecords”, “MemoryRecordsBuilder”]

File ~\AppData\Lib\site-packages\kafka\record\memory_records.py:27 25 from kafka.errors import CorruptRecordException 26 from kafka.record.abc import ABCRecords —> 27 from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder 28 from kafka.record.default_records import DefaultRecordBatch, DefaultRecordBatchBuilder 31 class MemoryRecords(ABCRecords):

File ~\AppData\Lib\site-packages\kafka\record\legacy_records.py:50 47 from kafka.record.abc import ABCRecord, ABCRecordBatch, ABCRecordBatchBuilder 48 from kafka.record.util import calc_crc32 —> 50 from kafka.codec import ( 51 gzip_encode, snappy_encode, lz4_encode, lz4_encode_old_kafka, 52 gzip_decode, snappy_decode, lz4_decode, lz4_decode_old_kafka, 53 ) 54 import kafka.codec as codecs 55 from kafka.errors import CorruptRecordException, UnsupportedCodecError

File ~\AppData\Lib\site-packages\kafka\codec.py:9 6 import struct 8 from kafka.vendor import six ----> 9 from kafka.vendor.six.moves import range 11 _XERIAL_V1_HEADER = (-126, b’S’, b’N’, b’A’, b’P’, b’P’, b’Y’, 0, 1, 1) 12 _XERIAL_V1_FORMAT = ‘bccccccBii’

ModuleNotFoundError: No module named ‘kafka.vendor.six.moves’

Can we please reopen this bug because a critical bug like this should not be closed until the fix is released. I would even pin it after reopening it, just to remind everyone about it.

It’s a fair point, and since there’s a demand to reopen this, I’ll oblige!

+1 for issue being present on pthon 3.12.3

This project has been unmaintained for almost a year; as the README suggests you might have better luck using kafka-python-ng. As multiple persons have confirmed, this workaround works.

(please stop adding “+1” comments; it’s useless and you’re notifying 20+ accounts for nothing each time)

Actually, I’m working correctly with the Python 3.7 version. Greetings

The issue is about Python 3.12; Python versions up to 3.11.x are fine.

@csharplus, It is working.

@csharplus , can confirm, that your workaround helped me to resolve the issue with kafka-python:2.0.2

i meet this problem in python3.12 kafka-python 2.0.2,so how to solve this problem

Hi @wbarnha , The issue still persists and I am facing the same issue, can you pls provide an update when the fix will be rolloed out. Thanks

This really needs to be addressed or this project abandoned. Why is @wbarnha the only person working on it? Is there some issue getting contributors onboarded?

Issue still persists! I was in the very early stage of my project so moved further ahead with confluent-kafka package.

Or you can go with kafka-python-ng as suggested by maintainer which is a forked version of this repo.

And still does not work with Python 3.13 either

I also have the same issue

Thanks for it @wbarnha , I downgraded my python version and got it rectified.