sentencepiece: Cannot install sentencepiece with Python 3.9 on Windows

Currently adding Python 3.9 support for pytorch/text and ran into an issue installing sentencepiece for Python 3.9 on windows. (CircleCI logs)

  ERROR: Failed building wheel for sentencepiece
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\circleci\project\env\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-trvw9qva\\sentencepiece_6ae2202249f44bf5b7a3902ec8532c93\\setup.py'"'"'; __file__='"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-trvw9qva\\sentencepiece_6ae2202249f44bf5b7a3902ec8532c93\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\circleci\AppData\Local\Temp\pip-record-xi27zjv8\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\circleci\project\env\Include\sentencepiece'
         cwd: C:\Users\circleci\AppData\Local\Temp\pip-install-trvw9qva\sentencepiece_6ae2202249f44bf5b7a3902ec8532c93\
    Complete output (20 lines):
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.9
    creating build\lib.win-amd64-3.9\sentencepiece
    copying src\sentencepiece/__init__.py -> build\lib.win-amd64-3.9\sentencepiece
    copying src\sentencepiece/sentencepiece_model_pb2.py -> build\lib.win-amd64-3.9\sentencepiece
    copying src\sentencepiece/sentencepiece_pb2.py -> build\lib.win-amd64-3.9\sentencepiece
    running build_ext
    building 'sentencepiece._sentencepiece' extension
    creating build\temp.win-amd64-3.9
    creating build\temp.win-amd64-3.9\Release
    creating build\temp.win-amd64-3.9\Release\src
    creating build\temp.win-amd64-3.9\Release\src\sentencepiece
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\circleci\project\env\include -IC:\Users\circleci\project\env\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt /EHsc /Tpsrc/sentencepiece/sentencepiece_wrap.cxx /Fobuild\temp.win-amd64-3.9\Release\src/sentencepiece/sentencepiece_wrap.obj /MT /I..\build\root\include
    cl : Command line warning D9025 : overriding '/MD' with '/MT'
    sentencepiece_wrap.cxx
    src/sentencepiece/sentencepiece_wrap.cxx(2777): fatal error C1083: Cannot open include file: 'sentencepiece_processor.h': No such file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.27.29110\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2

This is a duplicate of #452, but no real solution to building from source seems to have come from that so I have opened a new issue

Is there a workaround for getting this dependency?

cc @taku910

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 5
  • Comments: 16 (2 by maintainers)

Most upvoted comments

Hello everyone,

I receive the same message with python 3.11.0 on Win 10. I also tried installing from source but I receive the same error message:

src/sentencepiece/sentencepiece_wrap.cxx(2822): fatal error C1083: Cannot open include file: ‘sentencepiece_processor.h’: No such file or directory

I didn’t downgrade Python, instead I modified the requirements.txt file and added the following version information: sentencepiece==0.1.99

I had the same issue and I followed the instructions to build from source here and they worked: https://github.com/google/sentencepiece/blob/master/python/README.md

The only difference is I was using CMake GUI and Visual Studio to generate a solution and build the INSTALL project it generated.

You can manually grab a windows cp39 wheel from here until the maintainers cut a new release: https://ci.appveyor.com/project/taku910/sentencepiece/build/job/djp9386oghmdunxg/artifacts