transformers: cannot import name 'COMMON_SAFE_ASCII_CHARACTERS'

System Info

macbook pro m2 with anaconda, python 3.9

I’m running transformers on an m1 mac and am getting the following error when I import from transformers import OwlViTProcessor, OwlViTForObjectDetection

File ~/opt/anaconda3/envs/nd1/lib/python3.9/site-packages/transformers/init.py:26 23 from typing import TYPE_CHECKING 25 # Check the dependencies satisfy the minimal versions required. —> 26 from . import dependency_versions_check 27 from .utils import ( 28 OptionalDependencyNotAvailable, 29 _LazyModule, (…) 42 logging, 43 ) 46 logger = logging.get_logger(name) # pylint: disable=invalid-name

File ~/opt/anaconda3/envs/nd1/lib/python3.9/site-packages/transformers/dependency_versions_check.py:36 33 if pkg in deps: 34 if pkg == “tokenizers”: 35 # must be loaded here, or else tqdm check may fail —> 36 from .utils import is_tokenizers_available … —> 10 from charset_normalizer.md import mess_ratio 11 from charset_normalizer.models import CharsetMatches, CharsetMatch 12 from warnings import warn

AttributeError: partially initialized module ‘charset_normalizer’ has no attribute ‘md__mypyc’ (most likely due to a circular import)

Who can help?

No response

Information

The official example scripts
My own modified scripts

Tasks

An officially supported task in the examples folder (such as GLUE/SQuAD, …)
My own task or dataset (give details below)

Reproduction

Install transformers on a macbook m2

from transformers import OwlViTProcessor, OwlViTForObjectDetection

Expected behavior

it should import but instaed it gives the above error message

About this issue

Original URL
State: closed
Created a year ago
Comments: 17 (1 by maintainers)

Most upvoted comments

System Info

macbook air m2 with anaconda, python 3.9

I got a similar bug 🐛 ImportError: cannot import name 'COMMON_SAFE_ASCII_CHARACTERS' from 'charset_normalizer.constant'

When I encountered this I used:

pip install chardet

+113

tomo-cps on Mar 20, 2023

python -m pip install charset-normalizer==2.1.0

solves the issue

+34

IsraelAbebe on Mar 23, 2023

Actually I was getting the same error when importing OpenAI but that was solved by upgrading the package

pip install --upgrade chardet

And that worked!

AayushSameerShah on Aug 28, 2023

System Info

macbook m2 with anaconda, python 3.9

I got a similar bug 🐛 ImportError: cannot import name 'COMMON_SAFE_ASCII_CHARACTERS' from 'charset_normalizer.constant'

When I encountered this I used:
pip install chardet

Encountered the same error message when importing transformers. Installing chardet solved the issue.

Output for transformers-cli env

transformers version: 4.28.0.dev0
Platform: Linux-4.18.0-425.13.1.el8_7.x86_64-x86_64-with-glibc2.28
Python version: 3.9.16
Huggingface_hub version: 0.13.2
PyTorch version (GPU?): 1.13.1 (False)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?: no
Using distributed or parallel set-up in script?: no

ani0075saha on Mar 19, 2023

>>> from requests import HTTPError
Traceback (most recent call last):
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/requests/compat.py", line 11, in <module>
    import chardet
ModuleNotFoundError: No module named 'chardet'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/requests/__init__.py", line 45, in <module>
    from .exceptions import RequestsDependencyWarning
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/requests/exceptions.py", line 9, in <module>
    from .compat import JSONDecodeError as CompatJSONDecodeError
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/requests/compat.py", line 13, in <module>
    import charset_normalizer as chardet
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/charset_normalizer/__init__.py", line 23, in <module>
    from charset_normalizer.api import from_fp, from_path, from_bytes, normalize
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/charset_normalizer/api.py", line 10, in <module>
    from charset_normalizer.md import mess_ratio
  File "charset_normalizer/md.py", line 5, in <module>
ImportError: cannot import name 'COMMON_SAFE_ASCII_CHARACTERS' from 'charset_normalizer.constant' (/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/charset_normalizer/constant.py)

>>> from requests import Response
Traceback (most recent call last):
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/requests/compat.py", line 11, in <module>
    import chardet
ModuleNotFoundError: No module named 'chardet'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/requests/__init__.py", line 45, in <module>
    from .exceptions import RequestsDependencyWarning
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/requests/exceptions.py", line 9, in <module>
    from .compat import JSONDecodeError as CompatJSONDecodeError
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/requests/compat.py", line 13, in <module>
    import charset_normalizer as chardet
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/charset_normalizer/__init__.py", line 23, in <module>
    from charset_normalizer.api import from_fp, from_path, from_bytes, normalize
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/charset_normalizer/api.py", line 10, in <module>
    from charset_normalizer.md import mess_ratio
AttributeError: partially initialized module 'charset_normalizer' has no attribute 'md__mypyc' (most likely due to a circular import)

(huggingface-bug-test) anisaha1:~$ conda list 
# packages in environment at /<redacted>/anaconda3/envs/huggingface-bug-test:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
blas                      1.0                         mkl  
brotlipy                  0.7.0           py39h27cfd23_1003  
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2023.01.10           h06a4308_0  
certifi                   2022.12.7        py39h06a4308_0  
cffi                      1.15.1           py39h5eee18b_3  
charset-normalizer        3.1.0                    pypi_0    pypi
cryptography              39.0.1           py39h9ce1e76_0  
cuda-cudart               11.7.99                       0    nvidia
cuda-cupti                11.7.101                      0    nvidia
cuda-libraries            11.7.1                        0    nvidia
cuda-nvrtc                11.7.99                       0    nvidia
cuda-nvtx                 11.7.91                       0    nvidia
cuda-runtime              11.7.1                        0    nvidia
ffmpeg                    4.3                  hf484d3e_0    pytorch
filelock                  3.10.0                   pypi_0    pypi
flit-core                 3.6.0              pyhd3eb1b0_0  
freetype                  2.12.1               h4a9f257_0  
giflib                    5.2.1                h5eee18b_3  
gmp                       6.2.1                h295c915_3  
gnutls                    3.6.15               he1e5248_0  
huggingface-hub           0.13.3                   pypi_0    pypi
idna                      3.4              py39h06a4308_0  
intel-openmp              2021.4.0          h06a4308_3561  
jpeg                      9e                   h5eee18b_1  
lame                      3.100                h7b6447c_0  
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.38                 h1181459_1  
lerc                      3.0                  h295c915_0  
libcublas                 11.10.3.66                    0    nvidia
libcufft                  10.7.2.124           h4fbf590_0    nvidia
libcufile                 1.6.0.25                      0    nvidia
libcurand                 10.3.2.56                     0    nvidia
libcusolver               11.4.0.1                      0    nvidia
libcusparse               11.7.4.91                     0    nvidia
libdeflate                1.17                 h5eee18b_0  
libffi                    3.4.2                h6a678d5_6  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libiconv                  1.16                 h7f8727e_2  
libidn2                   2.3.2                h7f8727e_0  
libnpp                    11.7.4.75                     0    nvidia
libnvjpeg                 11.8.0.2                      0    nvidia
libpng                    1.6.39               h5eee18b_0  
libstdcxx-ng              11.2.0               h1234567_1  
libtasn1                  4.16.0               h27cfd23_0  
libtiff                   4.5.0                h6a678d5_2  
libunistring              0.9.10               h27cfd23_0  
libwebp                   1.2.4                h11a3e52_1  
libwebp-base              1.2.4                h5eee18b_1  
lz4-c                     1.9.4                h6a678d5_0  
mkl                       2021.4.0           h06a4308_640  
mkl-service               2.4.0            py39h7f8727e_0  
mkl_fft                   1.3.1            py39hd3c417c_0  
mkl_random                1.2.2            py39h51133e4_0  
ncurses                   6.4                  h6a678d5_0  
nettle                    3.7.3                hbbd107a_1  
numpy                     1.24.2                   pypi_0    pypi
numpy-base                1.23.5           py39h31eccc5_0  
openh264                  2.1.1                h4ff587b_0  
openssl                   1.1.1t               h7f8727e_0  
packaging                 23.0                     pypi_0    pypi
pillow                    9.4.0            py39h6a678d5_0  
pip                       23.0.1           py39h06a4308_0  
pycparser                 2.21               pyhd3eb1b0_0  
pyopenssl                 23.0.0           py39h06a4308_0  
pysocks                   1.7.1            py39h06a4308_0  
python                    3.9.16               h7a1cb2a_2  
pytorch                   1.13.1          py3.9_cuda11.7_cudnn8.5.0_0    pytorch
pytorch-cuda              11.7                 h778d358_3    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pyyaml                    6.0                      pypi_0    pypi
readline                  8.2                  h5eee18b_0  
regex                     2022.10.31               pypi_0    pypi
requests                  2.28.2                   pypi_0    pypi
setuptools                65.6.3           py39h06a4308_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.41.1               h5eee18b_0  
tk                        8.6.12               h1ccaba5_0  
tokenizers                0.13.2                   pypi_0    pypi
torchaudio                0.13.1               py39_cu117    pytorch
torchvision               0.14.1               py39_cu117    pytorch
tqdm                      4.65.0                   pypi_0    pypi
transformers              4.28.0.dev0              pypi_0    pypi
typing-extensions         4.5.0                    pypi_0    pypi
typing_extensions         4.4.0            py39h06a4308_0  
tzdata                    2022g                h04d1e81_0  
urllib3                   1.26.15                  pypi_0    pypi
wheel                     0.38.4           py39h06a4308_0  
xz                        5.2.10               h5eee18b_1  
zlib                      1.2.13               h5eee18b_0  
zstd                      1.5.2                ha4553b6_0

(huggingface-bug-test) anisaha1:~$ transformers-cli env
Traceback (most recent call last):
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/requests/compat.py", line 11, in <module>
    import chardet
ModuleNotFoundError: No module named 'chardet'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/bin/transformers-cli", line 5, in <module>
    from transformers.commands.transformers_cli import main
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/transformers/__init__.py", line 26, in <module>
    from . import dependency_versions_check
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/transformers/dependency_versions_check.py", line 17, in <module>
    from .utils.versions import require_version, require_version_core
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/transformers/utils/__init__.py", line 30, in <module>
    from .generic import (
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/transformers/utils/generic.py", line 29, in <module>
    from .import_utils import is_flax_available, is_tf_available, is_torch_available, is_torch_fx_proxy
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 32, in <module>
    from . import logging
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/transformers/utils/logging.py", line 35, in <module>
    import huggingface_hub.utils as hf_hub_utils
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/huggingface_hub/utils/__init__.py", line 32, in <module>
    from ._errors import (
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 3, in <module>
    from requests import HTTPError, Response
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/requests/__init__.py", line 45, in <module>
    from .exceptions import RequestsDependencyWarning
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/requests/exceptions.py", line 9, in <module>
    from .compat import JSONDecodeError as CompatJSONDecodeError
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/requests/compat.py", line 13, in <module>
    import charset_normalizer as chardet
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/charset_normalizer/__init__.py", line 23, in <module>
    from charset_normalizer.api import from_fp, from_path, from_bytes, normalize
  File "/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/charset_normalizer/api.py", line 10, in <module>
    from charset_normalizer.md import mess_ratio
  File "charset_normalizer/md.py", line 5, in <module>
ImportError: cannot import name 'COMMON_SAFE_ASCII_CHARACTERS' from 'charset_normalizer.constant' (/<redacted>/anaconda3/envs/huggingface-bug-test/lib/python3.9/site-packages/charset_normalizer/constant.py)

(huggingface-bug-test) anisaha1:~$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.7 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.7
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"

(base) anisaha1:~$ uname -r
4.18.0-425.13.1.el8_7.x86_64

https://github.com/psf/requests/issues/6384

ani0075saha on Mar 21, 2023

System Info

macbook air m2 with anaconda, python 3.9

I got a similar bug 🐛 ImportError: cannot import name 'COMMON_SAFE_ASCII_CHARACTERS' from 'charset_normalizer.constant'

When I encountered this I used:
pip install chardet

It’s working for me. You are the best.

bring-nirachornkul on Nov 21, 2023

I got the above error and did python -m pip install charset-normalizer==2.1.0. This gave me another error which went away after doing pip install chardet .

The error after 2.1.0 was as below but it was solved. I’m using M2 MAX and the packages below.

ImportError: cannot import name 'KO_NAMES' from 'charset_normalizer.constant' (/opt/anaconda3/envs/mlenv/lib/python3.8/site-packages/charset_normalizer/constant.py)

Package Version

anyio 3.6.2 appnope 0.1.2 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0 arrow 1.2.3 asttokens 2.0.5 attrs 22.2.0 backcall 0.2.0 beautifulsoup4 4.11.2 bleach 6.0.0 brotlipy 0.7.0 certifi 2022.12.7 cffi 1.15.1 chardet 5.1.0 charset-normalizer 2.1.0 click 8.1.3 comm 0.1.2 contourpy 1.0.7 cryptography 39.0.1 cycler 0.11.0 debugpy 1.6.6 decorator 5.1.1 defusedxml 0.7.1 executing 0.8.3 fastjsonschema 2.16.2 filelock 3.9.0 flit_core 3.6.0 fonttools 4.39.0 fqdn 1.5.1 future 0.18.2 gmpy2 2.1.2 huggingface-hub 0.12.1 idna 3.4 importlib-metadata 6.0.0 importlib-resources 5.12.0 ipykernel 6.21.2 ipython 8.10.0 ipython-genutils 0.2.0 ipywidgets 8.0.4 isoduration 20.11.0 jedi 0.18.1 Jinja2 3.1.2 joblib 1.2.0 jsonpointer 2.3 jsonschema 4.17.3 jupyter 1.0.0 jupyter_client 8.0.3 jupyter-console 6.6.1 jupyter_core 5.2.0 jupyter-events 0.6.3 jupyter_server 2.3.0 jupyter_server_terminals 0.4.4 jupyterlab-pygments 0.2.2 jupyterlab-widgets 3.0.5 kiwisolver 1.4.4 MarkupSafe 2.1.2 matplotlib 3.7.1 matplotlib-inline 0.1.6 mistune 2.0.5 mkl-fft 1.3.1 mkl-random 1.2.2 mkl-service 2.4.0 mpmath 1.3.0 nbclassic 0.5.2 nbclient 0.7.2 nbconvert 7.2.9 nbformat 5.7.3 nest-asyncio 1.5.6 networkx 3.0 nltk 3.8.1 notebook 6.5.2 notebook_shim 0.2.2 numpy 1.23.5 packaging 23.0 pandas 1.5.3 pandocfilters 1.5.0 parso 0.8.3 pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.4.0 pip 22.3.1 pkgutil_resolve_name 1.3.10 platformdirs 3.0.0 portalocker 2.7.0 prometheus-client 0.16.0 prompt-toolkit 3.0.36 psutil 5.9.4 ptyprocess 0.7.0 pure-eval 0.2.2 pycparser 2.21 Pygments 2.11.2 pyOpenSSL 23.0.0 pyparsing 3.0.9 pyrsistent 0.19.3 PySocks 1.7.1 python-dateutil 2.8.2 python-json-logger 2.0.7 pytorch-crf 0.7.2 pytz 2023.2 PyYAML 6.0 pyzmq 25.0.0 qtconsole 5.4.0 QtPy 2.3.0 regex 2022.10.31 requests 2.28.2 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 scikit-learn 1.2.2 scikit-plot 0.3.7 scipy 1.10.1 seaborn 0.12.2 Send2Trash 1.8.0 sentence-transformers 2.2.2 sentencepiece 0.1.97 setuptools 65.6.3 six 1.16.0 sklearn 0.0.post1 sniffio 1.3.0 soupsieve 2.4 stack-data 0.2.0 sympy 1.11.1 terminado 0.17.1 threadpoolctl 3.1.0 tinycss2 1.2.1 tokenizers 0.12.1 torch 2.0.0 torchaudio 2.0.0 torchdata 0.6.0 torchtext 0.13.0 torchvision 0.15.0 tornado 6.2 tqdm 4.64.1 traitlets 5.7.1 transformers 4.27.4 typing_extensions 4.4.0 uri-template 1.2.0 urllib3 1.26.15 wcwidth 0.2.5 webcolors 1.12 webencodings 0.5.1 websocket-client 1.5.1 wheel 0.38.4 widgetsnbextension 4.0.5 zipp 3.14.0

drei34 on Apr 5, 2023

I am currently having this issue with one anaconda environment and not another. This is very confusing.

jfjordanfarr on May 21, 2023

You can try opening an issue at charset_normalizer and point out that their 3.1.0 release seems broken on MacOS (but 2.1.0 works apparently, from the comment above).

From your traceback, the simple line import charset_normalizer should fail in your environment (it doesn’t in mine, but I’m not on MacOS).

sgugger on Mar 22, 2023

Yes, we can try to have it solve in requests first indeed. It’s if that takes too much time or is not deemed important we should fix it in hf hub.

@ani0075saha Could you try the two lines given by Wauplin and do step 2 and 3?

sgugger on Mar 20, 2023