sentencepiece: Pip install sentencepiece failure

For Apple M1 / ARM: Run these on default terminal (not sure if all packages are required):

brew install cmake
brew install gperftools
brew install pkg-config

Then run this on rosetta emulated terminal (maybe the default terminal will work too): pip3 install sentencepiece

I find a solution works for me.
Just: pip install --upgrade pip Env: python: 3.7.2 Release: 8.11 Codename: jessie

Just brew install cmake fixed the issue for me, thanks @ni9elf. Ran on default terminal window too

I come with this problem !

hardware: Apple Macbook M1 Python 3.9.4 (default, Apr 4 2021, 17:42:23) git clone https://github.com/google/sentencepiece.git cd python python3 setup.py install

logs as blew: /opt/homebrew/lib/python3.9/site-packages/setuptools/dist.py:642: UserWarning: Usage of dash-separated ‘description-file’ will not be supported in future versions. Please use the underscore name ‘description_file’ instead warnings.warn( running install running bdist_egg running egg_info writing src/sentencepiece.egg-info/PKG-INFO writing dependency_links to src/sentencepiece.egg-info/dependency_links.txt writing top-level names to src/sentencepiece.egg-info/top_level.txt reading manifest file ‘src/sentencepiece.egg-info/SOURCES.txt’ reading manifest template ‘MANIFEST.in’ writing manifest file ‘src/sentencepiece.egg-info/SOURCES.txt’ installing library code to build/bdist.macosx-11-arm64/egg running install_lib running build_py running build_ext Package sentencepiece was not found in the pkg-config search path. Perhaps you should add the directory containing sentencepiece.pc' to the PKG_CONFIG_PATH environment variable No package 'sentencepiece' found mkdir: bundled: File exists fatal: destination path 'sentencepiece' already exists and is not an empty directory. fatal: destination path 'sentencepiece' already exists and is not an empty directory. mkdir: build: File exists ./build_bundled.sh: line 15: cmake: command not found ./build_bundled.sh: line 16: nproc: command not found make: *** No targets specified and no makefile found. Stop. make: *** No rule to make target install’. Stop. Package sentencepiece was not found in the pkg-config search path. Perhaps you should add the directory containing `sentencepiece.pc’ to the PKG_CONFIG_PATH environment variable No package ‘sentencepiece’ found Failed to find sentencepiece pkg-config

None of this worked until I found this comment: https://github.com/google/sentencepiece/issues/608#issuecomment-1276909623

brew install sentencepiece

seems to have finally done it…

So

brew install cmake
brew install pkg-config
brew install sentencepiece
pip install sentencepiece

and then finally for Huggingface pip3 install 'transformers[tf-cpu]' to work, I had to also install Rust next…

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
pip3 install 'transformers[tf-cpu]'

Finally…!

One more data point for M1 users: I used the default terminal to run

brew install cmake
brew install pkg-config

And that did it for me.

conda install sentencepiece works for me

I don’t know which package helped me out. but after installing these packages my problem solved:

sudo apt-get install cmake build-essential pkg-config libgoogle-perftools-dev

It comes from the Build and install SentencePiece command line tools from C++ source section of the README.md

This is how I’ve fixed the problem on this particular server: Python 3.11.0, Ubuntu 20.04 (focal):

sudo apt-get install pkg-config
sudo apt-get install cmake

So basically I provided all necessary building tools to build sentencepiece from scratch, referring to @taku910 clarification: Seems the valid binary whl package is not downloaded and the fallback src package is fetched instead.

Personally, I didn’t want to depend on a boundary case for a specific scenario, but to keep generic installation

Same here, only happens on Python 3.7.4 for me, works on Python 3.6. Observed on Ubuntu 16.04.4

Collecting sentencepiece
  Downloading https://files.pythonhosted.org/packages/fd/45/6d0eb609d5cd81df094aab71a867b2ab6b315ffd592e78fb94a625c4d6aa/sentencepiece-0.1.3.tar.gz (498kB)
     |████████████████████████████████| 501kB 41.1MB/s
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Package sentencepiece was not found in the pkg-config search path.
    Perhaps you should add the directory containing `sentencepiece.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'sentencepiece' found
    Failed to find sentencepiece pkgconfig
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-cqmezxh3/sentencepiece/

One more data point for M1 users: I used the default terminal to run

brew install cmake
brew install pkg-config

And that did it for me.

Also on M1 - this worked for me

I confirm that installing cmake on Macos Ventura

brew install cmake

it allows me to install correctly sentencepiece python module

pip3 install  sentencepiece

Thank you!

I solved this first by running the below command

brew install sentencepiece

and then running pip install sentencepiece again.

None of this worked until I found this comment: #608 (comment)

brew install sentencepiece

seems to have finally done it…

So

brew install cmake
brew install pkg-config
brew install sentencepiece
pip install sentencepiece

and then finally for Huggingface pip3 install 'transformers[tf-cpu]' to work, I had to also install Rust next…

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
pip3 install 'transformers[tf-cpu]'

Finally…!

I have a M2 Macbook Pro and works after apply this thread!

Just want to add up probably might help. I switched to python version 3.9 and worked! I was actually at 3.10 when I did this and got those error.

pip3 install --upgrade pip it works for me

For anyone on Mac M1 this indeed fixed the issue:

I first tried:

pip install sentencepiece --use-pep517

didn’t work so then I did:

brew install cmake protobuf pkg-config
pip install sentencepiece

and indeed it worked 🎉🎉🎉

For Apple M1 / ARM: Run these on default terminal (not sure if all packages are required):

brew install cmake
brew install gperftools
brew install pkg-config

Then run this on rosetta emulated terminal (maybe the default terminal will work too): pip3 install sentencepiece

It worked solving the M2 issues too! Thanks, man!

I am Windows with Intel, but this also solved the issue for me: pip install cmake

My goal was to install FlairNLP, which relies on sentencepiece to run. After poking around at this problem for a while, this is what worked for me. I have a MacOS, I am using zsh, and I have Python 3.10.5. After activating my virtual environment, I ran the following command:

pip3 install --only-binary sentencepiece flair

I think this worked because the installation kept getting stuck on the step where it had to build a wheel for the sentencepiece dependency:

Building wheels for collected packages: sentencepiece Building wheel for sentencepiece (setup.py) … error error: subprocess-exited-with-error

So I read this article about Python wheels and figured I’d force pip to use wheels instead of source distributions for sentencepiece. Hope this helps!

#After installation cmake and pkg-config , I got this error

Package sentencepiece was not found in the pkg-config search path. Perhaps you should add the directory containing `sentencepiece.pc’ to the PKG_CONFIG_PATH environment variable No package ‘sentencepiece’ found Failed to find sentencepiece pkgconfig

@mprinc,

A very big THANK YOU!

Kind regards.

The pip install worked for a while but I got this

  Building wheel for sentencepiece (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [42 lines of output]
      /home/gelsyt/anaconda3/envs/contact_extraction_env/lib/python3.10/site-packages/setuptools/dist.py:757: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
        warnings.warn(
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-3.10/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.10/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.10/sentencepiece
      running build_ext
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      Cloning into 'sentencepiece'...
      Note: switching to '0e6dfbf86e2fa6d86a3d9a8a08a628da71c073e0'.
      
      You are in 'detached HEAD' state. You can look around, make experimental
      changes and commit them, and you can discard any commits you make in this
      state without impacting any branches by switching back to a branch.
      
      If you want to create a new branch to retain commits you create, you may
      do so (now or later) by using -c with the switch command. Example:
      
        git switch -c <new-branch-name>
      
      Or undo this operation with:
      
        git switch -
      
      Turn off this advice by setting config variable advice.detachedHead to false
      
      ./build_bundled.sh: 15: cmake: not found
      make: *** No targets specified and no makefile found.  Stop.
      make: *** No rule to make target 'install'.  Stop.
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      Failed to find sentencepiece pkg-config
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for sentencepiece
  Running setup.py clean for sentencepiece
Failed to build sentencepiece
Installing collected packages: tabulate, sqlitedict, sentencepiece, py4j, pptree, overrides, mpld3, janome, torch, segtok, pillow, networkx, more-itertools, lxml, langdetect, kiwisolver, joblib, importlib-metadata, future, ftfy, fonttools, deprecated, cycler, conllu, cloudpickle, wikipedia-api, scikit-learn, matplotlib, konoha, hyperopt, gensim, gdown, bpemb, flair
  Running setup.py install for sentencepiece ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for sentencepiece did not run successfully.
  │ exit code: 1
  ╰─> [30 lines of output]
      /home/gelsyt/anaconda3/envs/contact_extraction_env/lib/python3.10/site-packages/setuptools/dist.py:757: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
        warnings.warn(
      running install
      /home/gelsyt/anaconda3/envs/contact_extraction_env/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-3.10/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.10/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.10/sentencepiece
      running build_ext
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      mkdir: cannot create directory ‘bundled’: File exists
      fatal: destination path 'sentencepiece' already exists and is not an empty directory.
      fatal: destination path 'sentencepiece' already exists and is not an empty directory.
      mkdir: cannot create directory ‘build’: File exists
      ./build_bundled.sh: 15: cmake: not found
      make: *** No targets specified and no makefile found.  Stop.
      make: *** No rule to make target 'install'.  Stop.
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      Failed to find sentencepiece pkg-config
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> sentencepiece

Here was how I solved (partially). So I’m on Raspberry Pi 4 (armv7l GNU/Linux).

Installing Transformers, unable to install sentencepiece due to: “Failed to find sentencepiece pkgconfig”

I went deep into the cause, -------- First install from source: $ wget https://github.com/google/sentencepiece $ cd sentencepiece/python/ -------- Then try to install package (so folders are created): $ sudo python3 setup.py install -------- This will rise the error for me ERROR: Package sentencepiece was not found in the pkg-config search path. Perhaps you should add the directory containing `sentencepiece.pc’ to the PKG_CONFIG_PATH environment variable No package ‘sentencepiece’ found Failed to find sentencepiece pkgconfig" -------- excecute in terminal the individual set of commands $ cmake -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ … -DSPM_ENABLE_SHARED=OFF -DCMAKE_INSTALL_PREFIX=…/… $ make -stdlib=libstdc++ -j $(nproc) $ make install --------- once sentencepiece was installed $ python3 import sentencepiece ------- witch then rise a different error: Traceback (most recent call last): File “<stdin>”, line 1, in <module> File “/home/work/non.conflict/venv/lib/python3.7/site-packages/sentencepiece-0.1.94-py3.7-linux-armv7l.egg/sentencepiece/init.py”, line 13, in <module> from . import _sentencepiece ImportError: /home/work/non.conflict/venv/lib/python3.7/site-packages/sentencepiece-0.1.94-py3.7-linux-armv7l.egg/sentencepiece/_sentencepiece.cpython-37m-arm-linux-gnueabihf.so: undefined symbol: __atomic_fetch_add_8 --------- gives this error, but at now i’m able to import transformers import transformers