keops: pykeops broken on Arch Linux and Colab with CUDA 11

❯ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0

>>> import pykeops
>>> pykeops.clean_pykeops()          # just in case old build files are still present
Cleaning /home/scottgigante/.cache/pykeops-1.4.2-cpython-38/...
    - /home/scottgigante/.cache/pykeops-1.4.2-cpython-38/build-libKeOpstorch91904fe69f has been removed.
>>> pykeops.test_torch_bindings()    # perform the compilation
Compiling libKeOpstorch91904fe69f in /home/scottgigante/.cache/pykeops-1.4.2-cpython-38:
       formula: Sum_Reduction(SqNorm2(x - y),1)
       aliases: x = Vi(0,3); y = Vj(1,3);
       dtype  : float32
... cmake: /usr/lib/libc.so.6: version `GLIBC_2.33' not found (required by cmake)
cmake: /usr/lib/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/libuv.so.1)
cmake: /usr/lib/libc.so.6: version `GLIBC_2.33' not found (required by /usr/lib/libuv.so.1)

--------------------- CMAKE DEBUG -----------------
Command '['cmake', '/home/scottgigante/.local/lib/python3.8/site-packages/pykeops', "-DCMAKE_BUILD_TYPE='Release'", "-DFORMULA_OBJ='Sum_Reduction(SqNorm2(x - y),1)'", "-DVAR_ALIASES='auto x = Vi(0,3); auto y = Vj(1,3); '", "-Dshared_obj_name='libKeOpstorch91904fe69f'", "-D__TYPE__='float'", "-DPYTHON_LANG='torch'", "-DPYTHON_EXECUTABLE='/usr/bin/python'", "-DPYBIND11_PYTHON_VERSION='3.8'", '-DC_CONTIGUOUS=1', '-D__TYPEACC__=float', '-DSUM_SCHEME=1', '-DENABLECHUNK=1', '-DPYTORCH_ROOT_DIR=/home/scottgigante/.local/lib/python3.8/site-packages/torch', '-D_GLIBCXX_USE_CXX11_ABI=0', "-DcommandLine=cmake /home/scottgigante/.local/lib/python3.8/site-packages/pykeops -DCMAKE_BUILD_TYPE='Release' -DFORMULA_OBJ='Sum_Reduction(SqNorm2(x - y),1)' -DVAR_ALIASES='auto x = Vi(0,3); auto y = Vj(1,3); ' -Dshared_obj_name='libKeOpstorch91904fe69f' -D__TYPE__='float' -DPYTHON_LANG='torch' -DPYTHON_EXECUTABLE='/usr/bin/python' -DPYBIND11_PYTHON_VERSION='3.8' -DC_CONTIGUOUS=1 -D__TYPEACC__=float -DSUM_SCHEME=1 -DENABLECHUNK=1 -DPYTORCH_ROOT_DIR=/home/scottgigante/.local/lib/python3.8/site-packages/torch -D_GLIBCXX_USE_CXX11_ABI=0"]' returned non-zero exit status 1.

--------------------- ----------- -----------------
cmake: /usr/lib/libc.so.6: version `GLIBC_2.33' not found (required by cmake)
cmake: /usr/lib/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/libuv.so.1)
cmake: /usr/lib/libc.so.6: version `GLIBC_2.33' not found (required by /usr/lib/libuv.so.1)

--------------------- MAKE DEBUG -----------------
Command '['cmake', '--build', '.', '--target', 'libKeOpstorch91904fe69f', '--', 'VERBOSE=1']' returned non-zero exit status 1.

--------------------- ----------- -----------------
Done.
Compiling libKeOpstorch91904fe69f in /home/scottgigante/.cache/pykeops-1.4.2-cpython-38:
       formula: Sum_Reduction(SqNorm2(x - y),1)
       aliases: x = Vi(0,3); y = Vj(1,3);
       dtype  : float32
... cmake: /usr/lib/libc.so.6: version `GLIBC_2.33' not found (required by cmake)
cmake: /usr/lib/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/libuv.so.1)
cmake: /usr/lib/libc.so.6: version `GLIBC_2.33' not found (required by /usr/lib/libuv.so.1)

--------------------- CMAKE DEBUG -----------------
Command '['cmake', '/home/scottgigante/.local/lib/python3.8/site-packages/pykeops', "-DCMAKE_BUILD_TYPE='Debug'", "-DFORMULA_OBJ='Sum_Reduction(SqNorm2(x - y),1)'", "-DVAR_ALIASES='auto x = Vi(0,3); auto y = Vj(1,3); '", "-Dshared_obj_name='libKeOpstorch91904fe69f'", "-D__TYPE__='float'", "-DPYTHON_LANG='torch'", "-DPYTHON_EXECUTABLE='/usr/bin/python'", "-DPYBIND11_PYTHON_VERSION='3.8'", '-DC_CONTIGUOUS=1', '-D__TYPEACC__=float', '-DSUM_SCHEME=1', '-DENABLECHUNK=1', '-DPYTORCH_ROOT_DIR=/home/scottgigante/.local/lib/python3.8/site-packages/torch', '-D_GLIBCXX_USE_CXX11_ABI=0', "-DcommandLine=cmake /home/scottgigante/.local/lib/python3.8/site-packages/pykeops -DCMAKE_BUILD_TYPE='Debug' -DFORMULA_OBJ='Sum_Reduction(SqNorm2(x - y),1)' -DVAR_ALIASES='auto x = Vi(0,3); auto y = Vj(1,3); ' -Dshared_obj_name='libKeOpstorch91904fe69f' -D__TYPE__='float' -DPYTHON_LANG='torch' -DPYTHON_EXECUTABLE='/usr/bin/python' -DPYBIND11_PYTHON_VERSION='3.8' -DC_CONTIGUOUS=1 -D__TYPEACC__=float -DSUM_SCHEME=1 -DENABLECHUNK=1 -DPYTORCH_ROOT_DIR=/home/scottgigante/.local/lib/python3.8/site-packages/torch -D_GLIBCXX_USE_CXX11_ABI=0"]' returned non-zero exit status 1.

--------------------- ----------- -----------------
cmake: /usr/lib/libc.so.6: version `GLIBC_2.33' not found (required by cmake)
cmake: /usr/lib/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/libuv.so.1)
cmake: /usr/lib/libc.so.6: version `GLIBC_2.33' not found (required by /usr/lib/libuv.so.1)

--------------------- MAKE DEBUG -----------------
Command '['cmake', '--build', '.', '--target', 'libKeOpstorch91904fe69f', '--', 'VERBOSE=1']' returned non-zero exit status 1.

--------------------- ----------- -----------------
Done.
Traceback (most recent call last):
  File "/home/scottgigante/.local/lib/python3.8/site-packages/pykeops/test/install.py", line 57, in test_torch_bindings
    if torch.allclose(my_conv(x, y).view(-1), torch.tensor(expected_res).type(torch.float32)):
  File "/home/scottgigante/.local/lib/python3.8/site-packages/pykeops/torch/generic/generic_red.py", line 421, in __call__
    out = GenredAutograd.apply(self.formula, self.aliases, backend, self.dtype,
  File "/home/scottgigante/.local/lib/python3.8/site-packages/pykeops/torch/generic/generic_red.py", line 29, in forward
    myconv = LoadKeOps(formula, aliases, dtype, 'torch', optional_flags+include_dirs).import_module()
  File "/home/scottgigante/.local/lib/python3.8/site-packages/pykeops/common/keops_io.py", line 50, in import_module
    return importlib.import_module(self.dll_name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'libKeOpstorch91904fe69f'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/scottgigante/.local/lib/python3.8/site-packages/pykeops/test/install.py", line 68, in test_torch_bindings
    print(my_conv(x, y))
  File "/home/scottgigante/.local/lib/python3.8/site-packages/pykeops/torch/generic/generic_red.py", line 421, in __call__
    out = GenredAutograd.apply(self.formula, self.aliases, backend, self.dtype,
  File "/home/scottgigante/.local/lib/python3.8/site-packages/pykeops/torch/generic/generic_red.py", line 29, in forward
    myconv = LoadKeOps(formula, aliases, dtype, 'torch', optional_flags+include_dirs).import_module()
  File "/home/scottgigante/.local/lib/python3.8/site-packages/pykeops/common/keops_io.py", line 50, in import_module
    return importlib.import_module(self.dll_name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'libKeOpstorch91904fe69f'

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 23 (18 by maintainers)

Most upvoted comments

After some debugging work with @bcharlier, @joanglaunes and @gdurif, we have finally identified the problem:

/content/keops/pykeops/cmake_scripts/script_keops_formula/../../keops/core/formulas/maths/TensorDotNoTao.h(84): error: "auto" is not allowed here
/content/keops/pykeops/cmake_scripts/script_keops_formula/../../keops/core/formulas/maths/TensorDotNoTao.h(90): error: "auto" is not allowed here
2 errors detected in the compilation of "/content/keops/pykeops/keops/core/link_autodiff.cu".

Unfortunately, it seems that some of our advanced templating code to support the TensorDot (= “einsum”) operation is not handled well by nvcc 11 with gcc/g++ < 10. We will either fix the TensorDot implementation quickly, or disable this (fairly niche) operation on configurations that do not provide the latest gcc/g++. In any case, the problem should be fixed soon!

jeanfeydy on Mar 15, 2021

Hi @scottgigante , @annavaughan ,

Thanks a lot for your (critical) bug reports! New CUDA releases tend to break a few things here and there… We try to keep the compilation engine as simple as possible to avoid these problems, but they still happen sometimes. After checking in Google Colab, it seems that the bug is also present in our “master” branch: we’ll have to investigate a little bit.

Of course, we’re going to work on a fix as soon as possible and ideally release new packages by the end of the week. In the short term, a quick fix to get back to work is simply to downgrade CUDA. On Google Colab,

!apt-get install cuda=10.2.89-1
!pip install pykeops
import pykeops
pykeops.clean_pykeops()          # just in case old build files are still present
pykeops.test_numpy_bindings()    # perform the compilation

Should work fine.

We’ll come back to you soon! Best regards, Jean

jeanfeydy on Mar 3, 2021

Just ran into this issue so I thought I’d add to it.

c++ version 9.3.0 nvcc version 11.1

The problem was fixed by upgrading cmake from 3.16 to 3.18.

HOWEVER I was running in circles because for some reason upgrading cmake didn’t actually upgrade cmake at first. conda list showed cmake 3.18, but cmake --version showed 3.16 still. Checking which cmake returned the correct filepath (conda install for me).

I don’t know if restarting the terminal session fixed it, but for a while I was in a state where conda showed cmake 3.18 installed, but cmake --version was showing 3.16. If you run into this and aren’t able to fix it by upgrading cmake, make sure the cmake upgrade actually propagates.

kheyer on Oct 28, 2021

The v1.5 is now online on PyPi (thanks @bcharlier !). As detailed in the doc, the simplest way of using KeOps on Colab is now to install it with:

!pip install pykeops[colab]

This ensures that an up-to-date version of CMake is available. I’m closing the issue now - but feel free to re-open it if needed, especially @scottgigante if the problem remains on your configuration. Best regards, Jean

jeanfeydy on Mar 24, 2021