cudf: [DOC] [BUG] Building from source fails as deps are not fetched

Describe the bug Building v0.9.0 from source fails as some dependencies are missing or not fetched.

Steps/Code to reproduce bug

git clone and checkout v0.9.0.
update submodules
bash build.sh libcudf

-- RMM: RMM_LIBRARY set to RMM_LIBRARY-NOTFOUND
-- RMM: RMM_INCLUDE set to RMM_INCLUDE-NOTFOUND
-- DLPACK: DLPACK_INCLUDE set to DLPACK_INCLUDE-NOTFOUND
-- NVSTRINGS: NVSTRINGS_INCLUDE set to NVSTRINGS_INCLUDE-NOTFOUND
-- NVSTRINGS: NVSTRINGS_LIBRARY set to NVSTRINGS_LIBRARY-NOTFOUND
-- NVSTRINGS: NVCATEGORY_LIBRARY set to NVCATEGORY_LIBRARY-NOTFOUND
-- NVSTRINGS: NVTEXT_LIBRARY set to NVTEXT_LIBRARY-NOTFOUND

Expected behavior Build succeeds without missing deps.

Environment overview (please complete the following information)

Environment location: Centos 7, avx512
Method of cuDF install: source

Additional context

The documentation does not state dlpack, rmm or nvstrings as dependencies.
According to the RMM README

RMM currently must be built from source. This happens automatically in a submodule when you build or install cuDF or RAPIDS containers.

Users should then expect that theses deps should be automatically pulled.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 2
Comments: 42 (2 by maintainers)

Most upvoted comments

Hello @ccoulombe , I just installed CuDF without conda and as you said dfpack and rmm are missing.

Here’s my method to get it solved:

# Download dlpack from conda repo
mkdir /src/dlpack
cd /src/dlpack
wget https://anaconda.org/conda-forge/dlpack/0.2/download/linux-64/dlpack-0.2-he1b5a44_1.tar.bz2
tar xf dlpack-0.2-he1b5a44_1.tar.bz2
ln -s /src/dlpack $CUDF_HOME/cpp/thirdparty/dlpack

# Download dlpack from conda repo
mkdir /src/librmm
cd /src/librmm
wget https://anaconda.org/rapidsai-nightly/librmm/0.10.0a190925/download/linux-64/librmm-0.10.0a190925-cuda10.1_51.tar.bz2
tar xf librmm-0.10.0a190925-cuda10.1_51.tar.bz2

# Now run compilation
export DLPACK_ROOT=/src/dlpack
export RMM_ROOT=/src/librmm
cd $CUDF_HOME/cpp/build/
cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/lib/cudf/ -DCMAKE_CXX11_ABI=ON -DRMM_INCLUDE=/src/librmm/include/ -DDLPACK_INCLUDE=/src/dlpack/include/
make -j
make install

It works with:

Ubuntu 18.04
CUDA 10.1
CuDF : branch-0.10

ZuluPro on Oct 1, 2019

Ok thanks! It is important to specify that Arrow 2.0.0 does not work and it really need v1.0.0. It is written, but was down the doc page.

One thing that would be nice is to detail features that are require in Arrow and Pyarrow, for example, my installation of PyArrow has no IPC so the build failed trying to find the IPC header.

ccoulombe on Apr 23, 2021

Would suggest trying this on 0.19 as it’s now released. Building libcudf from source should handle all of its dependencies minus CUDA and Boost (Boost will be removed in 0.20 🎉), which need to be installed on the system in some way.

We’re still working on improving the experience in installing cuDF Python which has the challenge of depending on the PyArrow CUDA module which is not packaged via pip, but the rest of the dependencies should be able to be handled via a normal pip install at this point.

kkraus14 on Apr 22, 2021

With the merge of #7107 building libcudf from source should be infinitely easier. We can handle the entire dependency tree (other than Boost).

We still need to improve the Python side to clean it up a bit and improve the docs surrounding this, but we’re tackling it.

kkraus14 on Mar 5, 2021

@kkraus14 Thanks, I’ll give a deeper look.

Regarding your second comment. I totally agree, the deps then need to be clearly outline in the README.md.

Yes, I can certainly make such a PR!

ccoulombe on Nov 13, 2019