cuml: [BUG] Ridiculously huge effort needed for building cuML from source on a non-docker env!
Hi all, For the past 3 days (on-and-off), I’ve been trying to build cuML from source, outside of docker, and till today I haven’t succeeded in this effort at all!
So far… I’ve tried the following approaches. All of which have miserably failed.
conda + conda-cmake, but explicitly specified gcc
In this approach, I was using environmental modules with gcc v5.4.0, ctk v10.1.105, while cmake was installed via conda itself. This initially failed with FAISS complaining unable to find blas. Was able to fix this by installing openblas via conda itself. However, this fails at linker stage saying undefined reference to google::protobuf* messages. Even gcc v7.3.0 had the same issue.
conda + conda-cmake + conda-forge compilers
Same as above setup, but installed the compilers package from conda-forge channel. This now causes FAISS to error out saying unable to find cublas. When I look at config.log I noticed errors like undefined reference to cublasLt*. I could fix this issue locally via adding -lcublasLt option to the commandline, but couldn’t figure out how to pass this information from the cmake of cuML!
Expected behavior Atleast a guide/doc to setup a conflict-free environment for building cuML from source. A build.md is not enough here. Actually documenting a list of concrete steps would be better, IMO.
Environment details (please complete the following information):
- Environment location: Bare-metal
- Linux Distro/Architecture: CentOS 7
- GPU Model/Driver: V100 and driver 418.87
- CUDA: 10.1
- Method of cuDF install: conda
- Method of cuML install: source at HEAD of branch-0.12
- conda: installed via latest Miniconda script
Tagging @JohnZed @dantegd @cjnolet @datametrician
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 17 (16 by maintainers)
Hi all,
I spent about a week to build cuML without conda, docker, etc… Had a lot of bugs, but all of them were linker missing some libs. To fix protobuf bug mentioned above, I modified
CMakeFiles/cuml++.dir/link.txtfile and added-lprotobufat the end.Another common bug related to python packages is missing declaration of
cuda, cudartin setup.py For example, I modified setup.py this way to fix segfaul when I import cuml in python.libs = ['cuda', 'cudart', 'cuml++', 'rmm']library_dirs=[get_python_lib(), libcuml_path, "/usr/local/cuda-10.2/lib64/"]@teju85 Just jumping in here. The struggle has been that it’s a bit difficult to integrate
nvccinto the conda ecosystem and have it use the conda shipped host compilers reliably and easily. @jakirkham has been spearheading a lot of work in this space and I think we recently have come to a solution that is ready to have the tires kicked across the RAPIDS libraries.@dantegd here’s the output of print_env.sh
Click here to see environment details