kvikio: [Bug] Please enhance instruction for building Python package

Please update instruction or build script to make it work on any CUDA toolkit-enabled system.

Couldn’t find include header

❯ pyenv activate cucim-3.8
❯ cd python
❯ python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /home/gbae/repo/kvikio/python
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting cython
  Downloading Cython-0.29.28-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
     |████████████████████████████████| 1.9 MB 1.1 MB/s 
Building wheels for collected packages: kvikio
  Building wheel for kvikio (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/gbae/.pyenv/versions/cucim-3.8/bin/python /home/gbae/.pyenv/versions/cucim-3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp7ibfglvk
       cwd: /tmp/pip-req-build-d06p8fxv
  Complete output (27 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/kvikio
  copying kvikio/cufile.py -> build/lib.linux-x86_64-3.8/kvikio
  copying kvikio/zarr.py -> build/lib.linux-x86_64-3.8/kvikio
  copying kvikio/thread_pool.py -> build/lib.linux-x86_64-3.8/kvikio
  copying kvikio/_version.py -> build/lib.linux-x86_64-3.8/kvikio
  copying kvikio/__init__.py -> build/lib.linux-x86_64-3.8/kvikio
  creating build/lib.linux-x86_64-3.8/kvikio/_lib
  copying kvikio/_lib/__init__.py -> build/lib.linux-x86_64-3.8/kvikio/_lib
  copying kvikio/_lib/arr.pyi -> build/lib.linux-x86_64-3.8/kvikio/_lib
  UPDATING build/lib.linux-x86_64-3.8/kvikio/_version.py
  set build/lib.linux-x86_64-3.8/kvikio/_version.py to '0+unknown'
  running build_ext
  building 'kvikio._lib.libkvikio' extension
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/kvikio
  creating build/temp.linux-x86_64-3.8/kvikio/_lib
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -I/home/linuxbrew/.linuxbrew/opt/zlib -I/home/linuxbrew/.linuxbrew/opt/zlib -fPIC -O3 -I/home/gbae/.pyenv/versions/3.8.12/include -I/usr/local/cuda/include -I/home/gbae/.pyenv/versions/cucim-3.8/include -I/home/gbae/.pyenv/versions/3.8.12/include/python3.8 -c kvikio/_lib/libkvikio.cpp -o build/temp.linux-x86_64-3.8/kvikio/_lib/libkvikio.o -std=c++17
  kvikio/_lib/libkvikio.cpp:752:10: fatal error: kvikio/utils.hpp: No such file or directory
    752 | #include <kvikio/utils.hpp>
        |          ^~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for kvikio
Failed to build kvikio
ERROR: Could not build wheels for kvikio which use PEP 517 and cannot be installed directly
WARNING: You are using pip version 21.1.1; however, version 22.0.3 is available.
You should consider upgrading via the '/home/gbae/.pyenv/versions/cucim-3.8/bin/python -m pip install --upgrade pip' command.

Couldn’t find libnvidia-ml.so

❯ cd python
❯ python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /home/gbae/repo/kvikio/python
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting cython
  Downloading Cython-0.29.28-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
     |████████████████████████████████| 1.9 MB 679 kB/s 
Building wheels for collected packages: kvikio
  Building wheel for kvikio (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/gbae/.pyenv/versions/cucim-3.8/bin/python /home/gbae/.pyenv/versions/cucim-3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpio38_f81
       cwd: /tmp/pip-req-build-nnr151e7
  Complete output (62 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/kvikio
  copying kvikio/cufile.py -> build/lib.linux-x86_64-3.8/kvikio
  copying kvikio/zarr.py -> build/lib.linux-x86_64-3.8/kvikio
  copying kvikio/thread_pool.py -> build/lib.linux-x86_64-3.8/kvikio
  copying kvikio/_version.py -> build/lib.linux-x86_64-3.8/kvikio
  copying kvikio/__init__.py -> build/lib.linux-x86_64-3.8/kvikio
  creating build/lib.linux-x86_64-3.8/kvikio/_lib
  copying kvikio/_lib/__init__.py -> build/lib.linux-x86_64-3.8/kvikio/_lib
  copying kvikio/_lib/arr.pyi -> build/lib.linux-x86_64-3.8/kvikio/_lib
  UPDATING build/lib.linux-x86_64-3.8/kvikio/_version.py
  set build/lib.linux-x86_64-3.8/kvikio/_version.py to '0+unknown'
  running build_ext
  building 'kvikio._lib.libkvikio' extension
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/kvikio
  creating build/temp.linux-x86_64-3.8/kvikio/_lib
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -I/home/linuxbrew/.linuxbrew/opt/zlib -I/home/linuxbrew/.linuxbrew/opt/zlib -fPIC -O3 -I/home/gbae/.pyenv/versions/3.8.12/include -I/home/gbae/repo/kvikio/cpp/include -I/usr/local/cuda/include -I/home/gbae/.pyenv/versions/cucim-3.8/include -I/home/gbae/.pyenv/versions/3.8.12/include/python3.8 -c kvikio/_lib/libkvikio.cpp -o build/temp.linux-x86_64-3.8/kvikio/_lib/libkvikio.o -std=c++17
  kvikio/_lib/libkvikio.cpp:15862:20: warning: ‘__pyx_mdef___pyx_memoryviewslice_3__setstate_cython__’ defined but not used [-Wunused-variable]
  15862 | static PyMethodDef __pyx_mdef___pyx_memoryviewslice_3__setstate_cython__ = {"__setstate_cython__", (PyCFunction)__pyx_pw___pyx_memoryviewslice_3__setstate_cython__, METH_O, 0};
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  kvikio/_lib/libkvikio.cpp:15804:20: warning: ‘__pyx_mdef___pyx_memoryviewslice_1__reduce_cython__’ defined but not used [-Wunused-variable]
  15804 | static PyMethodDef __pyx_mdef___pyx_memoryviewslice_1__reduce_cython__ = {"__reduce_cython__", (PyCFunction)__pyx_pw___pyx_memoryviewslice_1__reduce_cython__, METH_NOARGS, 0};
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  kvikio/_lib/libkvikio.cpp:12959:20: warning: ‘__pyx_mdef___pyx_memoryview_3__setstate_cython__’ defined but not used [-Wunused-variable]
  12959 | static PyMethodDef __pyx_mdef___pyx_memoryview_3__setstate_cython__ = {"__setstate_cython__", (PyCFunction)__pyx_pw___pyx_memoryview_3__setstate_cython__, METH_O, 0};
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  kvikio/_lib/libkvikio.cpp:12901:20: warning: ‘__pyx_mdef___pyx_memoryview_1__reduce_cython__’ defined but not used [-Wunused-variable]
  12901 | static PyMethodDef __pyx_mdef___pyx_memoryview_1__reduce_cython__ = {"__reduce_cython__", (PyCFunction)__pyx_pw___pyx_memoryview_1__reduce_cython__, METH_NOARGS, 0};
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  kvikio/_lib/libkvikio.cpp:12807:20: warning: ‘__pyx_mdef_15View_dot_MemoryView_10memoryview_23copy_fortran’ defined but not used [-Wunused-variable]
  12807 | static PyMethodDef __pyx_mdef_15View_dot_MemoryView_10memoryview_23copy_fortran = {"copy_fortran", (PyCFunction)__pyx_memoryview_copy_fortran, METH_NOARGS, 0};
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  kvikio/_lib/libkvikio.cpp:12712:20: warning: ‘__pyx_mdef_15View_dot_MemoryView_10memoryview_21copy’ defined but not used [-Wunused-variable]
  12712 | static PyMethodDef __pyx_mdef_15View_dot_MemoryView_10memoryview_21copy = {"copy", (PyCFunction)__pyx_memoryview_copy, METH_NOARGS, 0};
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  kvikio/_lib/libkvikio.cpp:12635:20: warning: ‘__pyx_mdef_15View_dot_MemoryView_10memoryview_19is_f_contig’ defined but not used [-Wunused-variable]
  12635 | static PyMethodDef __pyx_mdef_15View_dot_MemoryView_10memoryview_19is_f_contig = {"is_f_contig", (PyCFunction)__pyx_memoryview_is_f_contig, METH_NOARGS, 0};
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  kvikio/_lib/libkvikio.cpp:12558:20: warning: ‘__pyx_mdef_15View_dot_MemoryView_10memoryview_17is_c_contig’ defined but not used [-Wunused-variable]
  12558 | static PyMethodDef __pyx_mdef_15View_dot_MemoryView_10memoryview_17is_c_contig = {"is_c_contig", (PyCFunction)__pyx_memoryview_is_c_contig, METH_NOARGS, 0};
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  kvikio/_lib/libkvikio.cpp:8685:20: warning: ‘__pyx_mdef___pyx_MemviewEnum_3__setstate_cython__’ defined but not used [-Wunused-variable]
   8685 | static PyMethodDef __pyx_mdef___pyx_MemviewEnum_3__setstate_cython__ = {"__setstate_cython__", (PyCFunction)__pyx_pw___pyx_MemviewEnum_3__setstate_cython__, METH_O, 0};
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  kvikio/_lib/libkvikio.cpp:8449:20: warning: ‘__pyx_mdef___pyx_MemviewEnum_1__reduce_cython__’ defined but not used [-Wunused-variable]
   8449 | static PyMethodDef __pyx_mdef___pyx_MemviewEnum_1__reduce_cython__ = {"__reduce_cython__", (PyCFunction)__pyx_pw___pyx_MemviewEnum_1__reduce_cython__, METH_NOARGS, 0};
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  kvikio/_lib/libkvikio.cpp:8071:20: warning: ‘__pyx_mdef___pyx_array_3__setstate_cython__’ defined but not used [-Wunused-variable]
   8071 | static PyMethodDef __pyx_mdef___pyx_array_3__setstate_cython__ = {"__setstate_cython__", (PyCFunction)__pyx_pw___pyx_array_3__setstate_cython__, METH_O, 0};
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  kvikio/_lib/libkvikio.cpp:8013:20: warning: ‘__pyx_mdef___pyx_array_1__reduce_cython__’ defined but not used [-Wunused-variable]
   8013 | static PyMethodDef __pyx_mdef___pyx_array_1__reduce_cython__ = {"__reduce_cython__", (PyCFunction)__pyx_pw___pyx_array_1__reduce_cython__, METH_NOARGS, 0};
        |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  g++ -pthread -shared -L/home/linuxbrew/.linuxbrew/opt/readline/lib -L/home/gbae/.pyenv/versions/3.8.12/lib -L/home/linuxbrew/.linuxbrew/lib -L/home/linuxbrew/.linuxbrew/opt/readline/lib -L/home/gbae/.pyenv/versions/3.8.12/lib -L/home/linuxbrew/.linuxbrew/lib build/temp.linux-x86_64-3.8/kvikio/_lib/libkvikio.o -L/home/gbae/.pyenv/versions/cucim-3.8/lib/python3.8/site-packages -L/usr/local/cuda/lib64 -lcuda -lnvidia-ml -lcufile -o build/lib.linux-x86_64-3.8/kvikio/_lib/libkvikio.cpython-38-x86_64-linux-gnu.so
  /home/linuxbrew/.linuxbrew/bin/ld: cannot find -lnvidia-ml
  collect2: error: ld returned 1 exit status
  error: command '/usr/bin/g++' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for kvikio

I workarounded the issue through the following change.

--- a/python/setup.py
+++ b/python/setup.py
@@ -45,8 +45,8 @@ this_setup_scrip_dir = os.path.dirname(os.path.realpath(__file__))
 
 # Throughout the script, we will populate `include_dirs`,
 # `library_dirs` and `depends`.
-include_dirs = [os.path.dirname(sysconfig.get_path("include"))]
-library_dirs = [get_python_lib()]
+include_dirs = [os.path.dirname(sysconfig.get_path("include"))] + ["/home/gbae/repo/kvikio/cpp/include"]
+library_dirs = [get_python_lib(), "/usr/local/cuda/lib64/stubs"]
 extra_objects = []
 depends = []  # Files to trigger rebuild when modified

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 20 (15 by maintainers)

Commits related to this issue

Merge pull request #12 from vyasr/chore/remove_unnecessary_ci Remove unnecessary workflows — committed to vuule/kvikio by shwina 9 months ago

Most upvoted comments

Yup, setup.py clean should fix that. I think this is an upstream issue in scikit-build, it looks similar but not exactly identical to https://github.com/scikit-build/scikit-build/issues/521

vyasr on Mar 31, 2022