hnswlib: pip install hnswlib fail on python 3.9

Hey all. I looked in the code and saw there’s a check for an environment variable. Try export HNSWLIB_NO_NATIVE=1 and try install again.

For windows:

1. Download Build Tools: https://visualstudio.microsoft.com/visual-cpp-build-tools/
2. Open, and then in Workloads select “Desktop development with C++” to install:

I have this problem with Python 3.10.3 on macOS 13.2.1 (M1)

...
        /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__memory/unique_ptr.h:610:45: error: unknown type name 'nullptr_t'
        operator!=(const unique_ptr<_T1, _D1>& __x, nullptr_t) _NOEXCEPT
                                                    ^
        fatal error: too many errors emitted, stopping now [-ferror-limit=]
        20 errors generated.
        error: command '/usr/bin/clang' failed with exit code 1
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed building wheel for hnswlib
  Failed to build hnswlib
  ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

I’m not using hnswlib directly, but it’s needed as a dependency of another package (https://github.com/chroma-core/chroma/issues/250)

Are there some instructions for how to install the python lib successfully?

UsingHNSWLIB_NO_NATIVE=1 I get a different error:

  Command ['/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10', '--no-deps', '/Users/anentropic/Library/Caches/pypoetry/artifacts/d2/32/ed/8674b10968a423c6f1b64821759e55a782ff3b28da34e2a886e85d54a9/hnswlib-0.7.0.tar.gz'] errored with the following return code 1, and output:
  Processing /Users/anentropic/Library/Caches/pypoetry/artifacts/d2/32/ed/8674b10968a423c6f1b64821759e55a782ff3b28da34e2a886e85d54a9/hnswlib-0.7.0.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    error: subprocess-exited-with-error

    × Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [16 lines of output]
        Traceback (most recent call last):
          File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
            main()
          File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
            return hook(config_settings)
          File "/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-29nj4h4p/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
            return self._get_build_requires(config_settings, requirements=['wheel'])
          File "/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-29nj4h4p/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
            self.run_setup()
          File "/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-29nj4h4p/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
            exec(code, locals())
          File "<string>", line 74, in <module>
          File "<string>", line 91, in BuildExt
        ValueError: list.remove(x): x not in list
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

Was getting the same error while installing on Linux inside Docker Container:

In file included from ./python_bindings/bindings.cpp:6:
      ./hnswlib/hnswlib.h: At global scope:
      ./hnswlib/hnswlib.h:80:13: warning: ‘bool AVX512Capable()’ defined but not used [-Wunused-function]
         80 | static bool AVX512Capable() {
            |             ^~~~~~~~~~~~~
      gcc: fatal error: Killed signal terminated program cc1plus
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

It revealed that problem was in low RAM (1GB). As I created swap-file for another 1GB everything worked just fine.

I used this commands to create SWAP-memory, if anyone needs:

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

And deleted it afterwards with

sudo swapoff /swapfile
sudo rm /swapfile

I have the same problem using python 3.9

Failed to build hnswlib ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

I have the same error on MacOS 13.1 and python 3.10. Setting the environment variable HNSWLIB_NO_NATIVE did not work for me. I do not have this problem on Windows 11.

same is happening as @anentropic mentioned I was installing chromadb and got a failure with python 3.9, any solutions people have found on this? which can help

For Linux users.

sudo apt-get install python3.11-dev

if chromadb is needed then sudo apt-get install chromadb

MACOS installation is fixed in the develop branch Could you try it? pip install git+https://github.com/nmslib/hnswlib.git@develop

This solution worked for me on Apple M1 with clang 14.0.3 and python 3.10.5. I had the same problem as previous reporters above. Did not need to set HNSWLIB_NO_NATIVE flag either. Thank you!

Installed version was:

pip freeze|grep hnsw
hnswlib @ git+https://github.com/nmslib/hnswlib.git@a4c8b0b9e33a813d94477e7eee79e50c04c42577 

This also makes it possible to install chromadb without any issues. (see https://github.com/chroma-core/chroma/issues/250)

Working on Windows 10 , and Python 3.12 , I’ve installed MSVC Tools and Windows 10 SDK and I’m still getting this error : Building wheel for chroma-hnswlib (pyproject.toml) … error error: subprocess-exited-with-error

× Building wheel for chroma-hnswlib (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [5 lines of output] running bdist_wheel running build building ‘hnswlib’ extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with “Microsoft C++ Build Tools”: https://visualstudio.microsoft.com/visual-cpp-build-tools/ [end of output]

choose these 2 components and ，it works well on windows11

I got this error along with pipenv even after setting up export HNSWLIB_NO_NATIVE=1

Python 3.11.2

Failed case😡

pipenv install chromadb

Worked case🥰

pipenv shell && pip install chromadb

export HNSWLIB_NO_NATIVE=1 export ARCHFLAGS=“-arch x86_64” and try again!!

Hi, Same problem here, python3.9 on macosx 13.3.1 export HNSWLIB_NO_NATIVE=1 from terminal before install works for me. Thanks 👍

I have the same problem using python 3.9

Failed to build hnswlib ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

any solution to fix that problem