faiss-wheels: faiss-gpu: index_cpu_to_gpu() hangs (doesn't occur with the conda package)
When I install faiss-gpu via pip, index_cpu_to_gpu() seems to hang forever. For example, this code sample hangs for me:
import faiss
index_flat = faiss.IndexFlatL2(16)
gpu_index_flat = faiss.index_cpu_to_gpu(faiss.StandardGpuResources(), 0, index_flat)
The index_cpu_to_gpu() will hang, spinning a CPU at 100% seemingly forever, mostly in libnvidia-ptxjitcompiler.so.510.54.
I confirmed this on two different machines with different GPUs (A100 and A10G).
I found two workarounds:
- Downgrade to faiss==1.5.3. I hit the problem with 1.6.0 or later.
- Use the conda package. I don’t observe the issue with conda, even with the latest version of faiss-gpu (1.7.2)
Maybe faiss-gpu wheel isn’t built with CUDA 11 support, and that’s why it doesn’t work with A100 / A10G?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 4
- Comments: 18 (7 by maintainers)
Hey everyone!
Thanks a lot for this chain! My issue got solved!
Issue: I was not able to use FAISS on the NVIDIA A100. The FAISS index was not getting pushed to the GPU!
For people who want a solution at one place. Please follow these steps below:
As faiss-gpu 1.7.3 is not available on pip. You can download the artifact directory (a directory containing multiple wheel files based on different systems) from here -> https://github.com/kyamagu/faiss-wheels/actions/runs/3487300515 . Go to bottom of the page -> locate artifacts and download it to your machine.
Once you have the artifacts folder, you need to see which file is the most suitable for you system. For example: I used the following file: faiss_gpu-1.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl . The filename itself indicates the configurations of a system. In the filename cp38 could mean CPython 3.8 (that is what I think). The artifacts directory also has many wheels for different systems. Locate the one compatible for your machine.
Once you have located your compatible wheel just run the following:
python -m pip install /path/to/wheel/faiss_gpu-1.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whlAnd you are good to go!
Thanks to @kyamagu for maintaining the repo!
https://github.com/kyamagu/faiss-wheels/releases/tag/v1.7.3
OK, yeah, that seems to work! (Not sure how that’s different from what I was trying previously.)
That wheel seems to fix my trivial repro, so it addresses the issue, at least as far as I can tell. Presumably this particular wheel won’t work for someone who has CUDA 10, but that’s a different problem. That’s why PyTorch versions are so complicated.
Check if the artifact in this CI run solves the issue. These wheels are built with CUDA 11.0.