kafka-python: "import kafka" fails with "ModuleNotFoundError: No module named 'kafka.vendor.six.moves'" under Python 3.12
I have verified this on Linux and Windows, even though the output below is from Linux:
user1@CY1-WL-337:~/dev$ python3.12 -m venv venv
user1@CY1-WL-337:~/dev$ source venv/bin/activate
(venv) user1@CY1-WL-337:~/dev$ pip install kafka-python
Collecting kafka-python
Using cached kafka_python-2.0.2-py2.py3-none-any.whl (246 kB)
Installing collected packages: kafka-python
Successfully installed kafka-python-2.0.2
(venv) user1@CY1-WL-337:~/dev$ python -c 'import kafka'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/__init__.py", line 23, in <module>
from kafka.consumer import KafkaConsumer
File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/__init__.py", line 3, in <module>
from kafka.consumer.group import KafkaConsumer
File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/group.py", line 13, in <module>
from kafka.consumer.fetcher import Fetcher
File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/consumer/fetcher.py", line 19, in <module>
from kafka.record import MemoryRecords
File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/__init__.py", line 1, in <module>
from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder
File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/memory_records.py", line 27, in <module>
from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder
File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/record/legacy_records.py", line 50, in <module>
from kafka.codec import (
File "/home/user1/dev/venv/lib/python3.12/site-packages/kafka/codec.py", line 9, in <module>
from kafka.vendor.six.moves import range
ModuleNotFoundError: No module named 'kafka.vendor.six.moves'
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 37 (6 by maintainers)
Commits related to this issue
- drop 3.12 because https://github.com/dpkp/kafka-python/issues/2412 — committed to haochenpan/diaspora-service by haochenpan 2 years ago
- Use kafka-python-ng instead of kafka-python Unfortunately kafka-python seems to be having maintenance issues: https://github.com/dpkp/kafka-python/issues/2431 And it cannot be installed on Python 3.... — committed to European-XFEL/DAMNIT by JamesWrigley 2 years ago
- Use kafka-python-ng instead of kafka-python Unfortunately kafka-python seems to be having maintenance issues: dpkp/kafka-python#2431 And it cannot be installed on Python 3.12: dpkp/kafka-python#2412 — committed to tavallaie/connectiva by tavallaie 2 years ago
This was patched in https://github.com/dpkp/kafka-python/pull/2398 and will be resolved in the next release. I apologize for the delay. Hopefully running
pip install git+https://github.com/dpkp/kafka-python.gitcould fulfill your needs in the meantime.Please understand that I am doing everything within my means to get a release out to the public.
The following code fix works for me with the current public build of
kafka-python:Can we please reopen this bug because a critical bug like this should not be closed until the fix is released. I would even pin it after reopening it, just to remind everyone about it.
This is still not fixed, and have been open for a year now. Does everyone really install from the github repo? Is it so hard to do a release?
Anyway, commenting to show additional demand. Although demand for “be able to use the library” seems self-explanatory…
The current workaround solution would be to replace your
kafka-pythoninstallation with my fork atkafka-python-ngfor now.Is there an idea of when the next release will be out?
Hey guys, this solved it for me: 1- Uninstall kafka-python using:
pip uninstall kafka-python2- Then run:pip install --break-system-packages git+https://github.com/dpkp/kafka-python.gitHi i am still having this issue. Is it resolved in the latest release.
here is my contect: Python 3.12.2 kafka-python 2.0.2
#import procedure: from kafka import KafkaProducer Error: ModuleNotFoundError Traceback (most recent call last) Cell In[7], line 1 ----> 1 from kafka import KafkaProducer
File ~\AppData\Lib\site-packages\kafka_init_.py:23 21 from kafka.admin import KafkaAdminClient 22 from kafka.client_async import KafkaClient —> 23 from kafka.consumer import KafkaConsumer 24 from kafka.consumer.subscription_state import ConsumerRebalanceListener 25 from kafka.producer import KafkaProducer
File ~\AppData\Lib\site-packages\kafka\consumer_init_.py:3 1 from future import absolute_import ----> 3 from kafka.consumer.group import KafkaConsumer 5 all = [ 6 ‘KafkaConsumer’ 7 ]
File ~\AppData\Lib\site-packages\kafka\consumer\group.py:13 10 from kafka.vendor import six 12 from kafka.client_async import KafkaClient, selectors —> 13 from kafka.consumer.fetcher import Fetcher 14 from kafka.consumer.subscription_state import SubscriptionState 15 from kafka.coordinator.consumer import ConsumerCoordinator
File ~\AppData\Lib\site-packages\kafka\consumer\fetcher.py:19 15 from kafka.protocol.fetch import FetchRequest 16 from kafka.protocol.offset import ( 17 OffsetRequest, OffsetResetStrategy, UNKNOWN_OFFSET 18 ) —> 19 from kafka.record import MemoryRecords 20 from kafka.serializer import Deserializer 21 from kafka.structs import TopicPartition, OffsetAndTimestamp
File ~\AppData\Lib\site-packages\kafka\record_init_.py:1 ----> 1 from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder 3 all = [“MemoryRecords”, “MemoryRecordsBuilder”]
File ~\AppData\Lib\site-packages\kafka\record\memory_records.py:27 25 from kafka.errors import CorruptRecordException 26 from kafka.record.abc import ABCRecords —> 27 from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder 28 from kafka.record.default_records import DefaultRecordBatch, DefaultRecordBatchBuilder 31 class MemoryRecords(ABCRecords):
File ~\AppData\Lib\site-packages\kafka\record\legacy_records.py:50 47 from kafka.record.abc import ABCRecord, ABCRecordBatch, ABCRecordBatchBuilder 48 from kafka.record.util import calc_crc32 —> 50 from kafka.codec import ( 51 gzip_encode, snappy_encode, lz4_encode, lz4_encode_old_kafka, 52 gzip_decode, snappy_decode, lz4_decode, lz4_decode_old_kafka, 53 ) 54 import kafka.codec as codecs 55 from kafka.errors import CorruptRecordException, UnsupportedCodecError
File ~\AppData\Lib\site-packages\kafka\codec.py:9 6 import struct 8 from kafka.vendor import six ----> 9 from kafka.vendor.six.moves import range 11 _XERIAL_V1_HEADER = (-126, b’S’, b’N’, b’A’, b’P’, b’P’, b’Y’, 0, 1, 1) 12 _XERIAL_V1_FORMAT = ‘bccccccBii’
ModuleNotFoundError: No module named ‘kafka.vendor.six.moves’
It’s a fair point, and since there’s a demand to reopen this, I’ll oblige!
+1 for issue being present on pthon 3.12.3
This project has been unmaintained for almost a year; as the README suggests you might have better luck using kafka-python-ng. As multiple persons have confirmed, this workaround works.
(please stop adding “+1” comments; it’s useless and you’re notifying 20+ accounts for nothing each time)
I am aware of the issue. Please refer to https://github.com/dpkp/kafka-python/issues/2412#issuecomment-1806341342.
The issue is about Python 3.12; Python versions up to 3.11.x are fine.
@csharplus, It is working.
@csharplus , can confirm, that your workaround helped me to resolve the issue with kafka-python:2.0.2
i meet this problem in python3.12 kafka-python 2.0.2,so how to solve this problem
Hi @wbarnha , The issue still persists and I am facing the same issue, can you pls provide an update when the fix will be rolloed out. Thanks
This really needs to be addressed or this project abandoned. Why is @wbarnha the only person working on it? Is there some issue getting contributors onboarded?
Issue still persists! I was in the very early stage of my project so moved further ahead with confluent-kafka package.
Or you can go with kafka-python-ng as suggested by maintainer which is a forked version of this repo.
And still does not work with Python 3.13 either
+1
I also have the same issue
Thanks for it @wbarnha , I downgraded my python version and got it rectified.