cudf: [BUG] unknown type in template argument, when building cudf python bindings on ARM
After a long struggle I have managed to install successfully libcudf, nvstrings and dependencies such as dlpack, rmm on Jetson Nano (ARM - AArch64).
I had completely functional libarrow and Pyarrow already in my machine - Pyarrow on ARM.
But, I’ve reached a dead end when building cudf python bindings due to a error wehn compiling cython arrow files.
Describe the bug
sudo -E python3 setup.py build_ext --inplace
Compiling cudf/_lib/arrow/_cuda.pyx because it changed.
Compiling cudf/_lib/avro.pyx because it changed.
Compiling cudf/_lib/binops.pyx because it changed.
Compiling cudf/_lib/concat.pyx because it changed.
Compiling cudf/_lib/copying.pyx because it changed.
Compiling cudf/_lib/csv.pyx because it changed.
Compiling cudf/_lib/cudf.pyx because it changed.
Compiling cudf/_lib/dlpack.pyx because it changed.
Compiling cudf/_lib/filling.pyx because it changed.
Compiling cudf/_lib/gpuarrow.pyx because it changed.
Compiling cudf/_lib/groupby.pyx because it changed.
Compiling cudf/_lib/hash.pyx because it changed.
Compiling cudf/_lib/issorted.pyx because it changed.
Compiling cudf/_lib/join.pyx because it changed.
Compiling cudf/_lib/json.pyx because it changed.
Compiling cudf/_lib/nvtx.pyx because it changed.
Compiling cudf/_lib/orc.pyx because it changed.
Compiling cudf/_lib/parquet.pyx because it changed.
Compiling cudf/_lib/quantile.pyx because it changed.
Compiling cudf/_lib/reduce.pyx because it changed.
Compiling cudf/_lib/replace.pyx because it changed.
Compiling cudf/_lib/reshape.pyx because it changed.
Compiling cudf/_lib/rolling.pyx because it changed.
Compiling cudf/_lib/search.pyx because it changed.
Compiling cudf/_lib/sort.pyx because it changed.
Compiling cudf/_lib/stream_compaction.pyx because it changed.
Compiling cudf/_lib/table.pyx because it changed.
Compiling cudf/_lib/transpose.pyx because it changed.
Compiling cudf/_lib/typecast.pyx because it changed.
Compiling cudf/_lib/unaryops.pyx because it changed.
Compiling cudf/_lib/utils.pyx because it changed.
Compiling cudf/_libxx/column.pyx because it changed.
Compiling cudf/_libxx/copying.pyx because it changed.
Compiling cudf/_libxx/stream_compaction.pyx because it changed.
Compiling cudf/_libxx/table.pyx because it changed.
[ 1/35] Cythonizing cudf/_lib/arrow/_cuda.pyx
Error compiling Cython file:
------------------------------------------------------------
...
"""
def __cinit__(self, CudaBuffer obj):
self.buffer = obj
self.reader = new CCudaBufferReader(self.buffer.buffer)
self.set_random_access_file(
shared_ptr[RandomAccessFile](self.reader))
^
------------------------------------------------------------
cudf/_lib/arrow/_cuda.pyx:740:23: unknown type in template argument
Error compiling Cython file:
------------------------------------------------------------
...
buffering.
"""
def __cinit__(self, CudaBuffer buffer):
self.buffer = buffer
self.writer = new CCudaBufferWriter(self.buffer.cuda_buffer)
self.set_output_stream(shared_ptr[OutputStream](self.writer))
^
------------------------------------------------------------
cudf/_lib/arrow/_cuda.pyx:787:42: unknown type in template argument
Traceback (most recent call last):
File "setup.py", line 76, in <module>
ext_modules=cythonize(extensions),
File "/home/abishek/.local/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1096, in cythonize
cythonize_one(*args)
File "/home/abishek/.local/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1219, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: cudf/_lib/arrow/_cuda.pyx
Steps/Code to reproduce bug Build cudf python bindings on ARM after building and installing libcudf.
Expected behavior Successful compilation of cudf python bindings.
Environment overview (please complete the following information)
- Environment location: Bare-metal.
- Method of cuDF install: from source.
Environment details
Click here to see environment details
**git*** commit d6b4794b4a3ed7e6577042596a32732bd62fd074 (HEAD -> branch-0.13, origin/branch-0.13, origin/HEAD) Merge: ecfd1307a a0c64d997 Author: Raza Jafri <razajafri@users.noreply.github.com> Date: Sat Feb 8 21:39:01 2020 -0800 Merge pull request #4069 from razajafri/stringcast Added cast of numeric columns to/from string **git submodules*** b165e1fb11eeea64ccf95053e40f2424312599cc thirdparty/cub (v1.7.1) bcd545071c7a5ddb28cb6576afc6399eb1286c43 thirdparty/jitify (heads/cudf) cdcda484d0c7db114ea29c3b33429de5756ecfd8 thirdparty/libcudacxx (0.8.1-99-gcdcda48) a97a7380c76346c22bb67b93695bed19592afad2 thirdparty/libcudacxx/libcxx (heads/rapidsai-interop) ***OS Information*** DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS" NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.3 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic Linux Titan 4.9.140-tegra #1 SMP PREEMPT Sat Oct 19 15:54:06 PDT 2019 aarch64 aarch64 aarch64 GNU/Linux ***GPU Information***./print_env.sh: line 25: nvidia-smi: command not found
***CPU*** Architecture: aarch64 Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Vendor ID: ARM Model: 1 Model name: Cortex-A57 Stepping: r1p1 CPU max MHz: 1428.0000 CPU min MHz: 102.0000 BogoMIPS: 38.40 L1d cache: 32K L1i cache: 48K L2 cache: 2048K Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 ***CMake*** /usr/local/bin/cmake cmake version 3.16.4 CMake suite maintained and supported by Kitware (kitware.com/cmake). ***g++*** /usr/bin/g++ g++ (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04.1) 7.4.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** /usr/local/cuda/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Mon_Mar_11_22:13:24_CDT_2019 Cuda compilation tools, release 10.0, V10.0.326 ***Python*** /usr/bin/python Python 2.7.15 ***Environment Variables*** PATH : /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/abishek/.local/bin:/usr/local/cuda/bin:/usr/local/cuda/bin LD_LIBRARY_PATH : /usr/local/lib::/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64 NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : PYTHON_PATH : conda not found pip not found
Additional context Although env reports default Python as 2.7.15, I’ve used Python3 throughout my build. If this bug is resolved, I expect a successful cudf installation on ARM.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 27
@heavyinfo You could try passing an empty string to the
GPU_ARCHScmake variable when building libcudf: https://github.com/rapidsai/cudf/blob/branch-0.13/cpp/CMakeLists.txt#L83That should try to build it for compute 5.3 but unsure how compilation / execution will go.