transformers: Installation Error - Failed building wheel for tokenizers
š Bug
Information
Model I am using (Bert, XLNet ā¦): N/A
Language I am using the model on (English, Chinese ā¦): N/A
The problem arises when using:
- the official example scripts: (give details below)
Problem arises in transformers installation on Microsoft Windows 10 Pro, version 10.0.17763
After creating and activating the virtual environment, installing transformers is not possible, because the following error occurs:
āerror: can not find Rust Compilerā āERROR: Failed building wheel for tokenizersā Failed to build tokenizers ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed d
The tasks I am working on is: [X ] transformers installation
To reproduce
Steps to reproduce the behavior:
- From command line interface, create and activate a virtual environment by following the steps in this URL: https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/
- Install transformers from source, by following the example in the topic From Source on this URL: https://github.com/huggingface/transformers
-m pip --version
-m pip install --upgrade pip
-m pip install --user virtualenv
-m venv env
.\env\Scripts\activate
pip install transformers
ERROR: Command errored out with exit status 1:
command: 'c:\users\vbrandao\env\scripts\python.exe' 'c:\users\vbrandao\env\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\vbrandao\AppData\Local\Temp\tmpj6evjmze'
cwd: C:\Users\vbrandao\AppData\Local\Temp\pip-install-sza2_lmj\tokenizers
Complete output (10 lines):
running bdist_wheel
running build
running build_py
creating build
creating build\lib
creating build\lib\tokenizers
copying tokenizers\__init__.py -> build\lib\tokenizers
running build_ext
running build_rust
error: Can not find Rust compiler
----------------------------------------
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly
Expected behavior
Installation of transformers should be complete.
Environment info
transformersversion: N/A - installation step- Platform: Command Line Interface / Virtual Env
- Python version: python 3.8
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): N/A
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 76
- Comments: 61 (2 by maintainers)
Links to this issue
Commits related to this issue
- Update required tokenizers version in setup.py to 0.6.0 as suggested in https://github.com/huggingface/transformers/issues/2831 — committed to lkemkes/transformers by lkemkes 4 years ago
I managed to solve the issue by installing Rust compiler
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | shpip install transformers==2.5.1I was having the same issue on virtualenv over Mac OS Mojave. Managed to solve it and install Transformers 2.5.1 by manually install the last version of tokenizers (0.6.0) instead of 0.5.2 that is required in the transformer package.
pip install tokenizers
Git clone latest version of transformers:
git clone https://github.com/huggingface/transformers
Before running the installation edit transformers/setup.py and change requirement of tokenizers to 0.6.0
Line 93: install_requires=[ ānumpyā, ātokenizers == 0.6.0ā,
Then run as usual:
cd transformers pip install .
I assume that you could also skip the first step and just collect the package as you run the install. Iām quite new to this, so just wanted to share my take.
Environment: macOS Mojave Ver 10.14.6 Tried installing both from pip and source. Same issue:
Result was that Transformers was not installed (not listed in pip freeze)
This however should work - seems like you just wonāt get the the new tokenizers: pip install transformers==2.4.1
Hi, I had the same problem and resolved it by installing rust. āerror: Can not find Rust compilerā
For MacOS, I used ācurl https://sh.rustup.rs -sSf | shā. I also found that it needed a nightly version of rust, so you have to specify that in the install options.
It works for me, thanks! You can do
source $HOME/.cargo/envinstead of restarting the terminal.Having the exact same issue on a Linux machine!
Installing Rust via
homebrewdid it for me on a Mac OS Monterey M1 SiliconUsing a conda environmentā¦
sudo apt install rustc cargosolved my problem finallyconda install -c huggingface transformers
Use This it will work for sure (M1 also) no need for rust if u get sure try rust and then this in your specific env
I am facing a similar issue trying to build on a PowerPC with RedHat I am getting errors when trying to build tokenizers:
Hi, I also had the same problem with my initial installation of the library. After some time, I realized that my anaconda version was on 32Bit. You can check your version with
python -c "import struct;print( 8 * struct.calcsize('P'))"The output should be 64. If it is 32 then you have to reinstall your IDEUsing an M1 macbook here. This solved the issue for me, thanks a ton!
@ArthurZucker that is ok, when transformers==4.37.2,tokenizers is fixed!!
@gardnerds also worked for me. Using python 3.7 and built from source using a clean conda env
could you try to update to the latest tokenizers version? We published a release yesterday!
Mac M1 Monterry.
pip install transformers==**does not work for my site.conda install transformersworks well.Rust compiler, fixed this for me.
curl --proto ā=httpsā --tlsv1.2 -sSf https://sh.rustup.rs | sh
Solved this problem by using python=3.7.9 instead of python=3.6.7 in conda env
@gardnerds, adding
$HOME/.cargo/binto PATH after installing rust fixed my installation. Thank you.@Wild3d I can confirm after running your snippet that I am on a 64bit version
I had this error when trying to install a specific version of
transformers:pip install transformers==4.15.0When I tried to install without specifying the version, it worked:pip install transformersI had the same problem when trying to create a dockerfile. I was able to solve using the solution of this issue.
Same issue, missing Rust compiler. This command fixed it
Only solution you need
Successfully built tokenizers Failed to build sentencepiece Installing collected packages: tokenizers, sentencepiece, certifi, urllib3, tqdm, regex, jmespath, idna, filelock, click, charset-normalizer, sacremoses, requests, botocore, s3transfer, boto3, transformers Running setup.py install for sentencepiece ā¦ error
I have the same python version and the exactly same question. I am running on Mac 2. I tried to lower the version of transformers and rustc. Both did not work out.
I have same problem how to solve?
I have installed rust from here and now have different error
windows 10
Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] on win32
https://github.com/xashru/punctuation-restoration/blob/master/requirements.txt
https://www.rust-lang.org/tools/install
Yes @bfelbo, I tried the below in requirements.txt and it worked but couldnāt use it as Vercel have hard dependency of 150MB build.
requirements.txt
Iām facing the issue in Vercel serverless deployments, and the problem is Iācant install rust there? Any other solution? Python version is 3.9
@AvivNavon Thanks ! Solved my problem too. (MacOS Mojave) I install latest version of transformers though (2.8.0)
pip install transformersinstead ofpip install transformers==2.5.1