cupy: Warn if nvcc found in PATH does not match with `$CUDA_PATH/bin/nvcc`

Conditions (you can just paste the output of python -c 'import cupy; cupy.show_config()')
- CuPy version github repo 7f24cb5a5
- OS/Platform CentOS 7.8.2003
- CUDA version 11.0
- cuDNN/NCCL version (if applicable)
Code to reproduce

export LC_ALL="en_US.UTF-8"
export CUDA_PATH=$LINUXBREWHOME/Cellar/cuda/11.0
export CC=$LINUXBREWHOME/opt/gcc@9/bin/gcc-9
export CXX=$LINUXBREWHOME/opt/gcc@9/bin/g++-9 
export LIBRARY_PATH="${CUDA_PATH}/lib"
export LD_LIBRARY_PATH="${CUDA_PATH}/lib:${LD_LIBRARY_PATH}"
export NVCC="nvcc --compiler-bindir $LINUXBREWHOME/opt/gcc@9/bin"
pip install . -vvvv

Error messages, stack traces, or logs

    building 'cupy.cuda.nvtx' extension
    //home/aznb/.pyenv/versions/3.8.3/include/python3.8 -c cupy/cuda/nvtx.cpp -o build/temp.linux-x86_64-3.8/cupy/cuda/nvtx.o -fopenmp
    //home/aznb/.linuxbrew/Cellar/cuda/11.0/lib
    building 'cupy_backends.cuda.libs.cutensor' extension
    //home/aznb/.pyenv/versions/3.8.3/include/python3.8 -c cupy_backends/cuda/libs/cutensor.cpp -o build/temp.linux-x86_64-3.8/cupy_backends/cuda/libs/cutensor.o -fopenmp
    //home/aznb/.linuxbrew/Cellar/cuda/11.0/lib
    building 'cupy.cuda.cub' extension
    //home/aznb/.pyenv/versions/3.8.3/include/python3.8 -c cupy/cuda/cub.cpp -o build/temp.linux-x86_64-3.8/cupy/cuda/cub.o -fopenmp
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/setup.py", line 169, in <module>
        setup(
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/cupy_setup_build.py", line 1012, in run
        build_ext.build_ext.run(self)
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 84, in run
        _build_ext.run(self)
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
        _build_ext.build_ext.run(self)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
        _build_ext.build_ext.build_extensions(self)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 205, in build_extension
        _build_ext.build_extension(self, ext)
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
        objects = self.compiler.compile(sources,
      File "//home/aznb/.pyenv/versions/3.8.3/lib/python3.8/distutils/ccompiler.py", line 574, in compile
        self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/cupy_setup_build.py", line 868, in _compile
        base_opts = build.get_compiler_base_options()
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/install/build.py", line 232, in get_compiler_base_options
        _compiler_base_options = _get_compiler_base_options()
      File "/tmp/160405612.1.noble-long.q/pip-req-build-lj1opkee/install/build.py", line 270, in _get_compiler_base_options
        stderrdata.decode('utf8'))
    UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 4-5: invalid continuation byte
    Running setup.py install for cupy: finished with status 'error'
ERROR: Command errored out with exit status 1: //home/aznb/.pyenv/versions/3.8.3/envs/loos/include/site/python3.8/cupy Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 827, in install
    success = install_legacy(
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/operations/install/legacy.py", line 86, in install
    raise LegacyInstallFailure
pip._internal.operations.install.legacy.LegacyInstallFailure

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 398, in run
    installed = install_given_reqs(
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 67, in install_given_reqs
    requirement.install(
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 845, in install
    six.reraise(*exc.parent)
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_vendor/six.py", line 703, in reraise
    raise value
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/operations/install/legacy.py", line 74, in install
    runner(
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/utils/subprocess.py", line 270, in runner
    call_subprocess(
  File "//home/aznb/.pyenv/versions/3.8.3/envs/loos/lib/python3.8/site-packages/pip/_internal/utils/subprocess.py", line 241, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: //home/aznb/.pyenv/versions/3.8.3/envs/loos/include/site/python3.8/cupy Check the logs for full command output.
Removed build tracker: '/tmp/160405612.1.noble-long.q/pip-req-tracker-rwqitlud'

About this issue

Original URL
State: open
Created 4 years ago
Comments: 24 (14 by maintainers)

Most upvoted comments

What actually works export NVCC="${CUDA_PATH}/bin/nvcc --compiler-bindir $LINUXBREWHOME/opt/gcc@9/bin" which seems like a bug in the build config because ${CUDA_PATH}/bin is actually in my $PATH and the build system should just look up nvcc in ${CUDA_PATH}/bin anyway and should not risk allowing the user to set $NVCC to a different version of CUDA than ${CUDA_PATH}. Also, that nvcc is the only one I have on the system and I don’t see how it would complain about missing header given all the path set up the way in the OP

Char-Aznable on Jul 15, 2020

btw @kmaehashi the title should be nvcc not nccl 😁

leofang on Jul 24, 2020

The error message is: b'cc1plus: fatal error: cuda_runtime.h: No such file or directory\ncompilation terminated.\n'

which is weird because earlier in the log, the cuda include path is set up correctly:

Include directories: ['/home/aznb/.linuxbrew/Cellar/cuda/11.0/include/cub', '/data/scratch/ssd/tmp/160441231.1.long.q/pip-req-build-tm1iq8f4/install/../cupy/core/include', '/home/aznb/.linuxbrew/Cellar/cuda/11.0/include']

and I can confirm that

ls -lh /home/aznb/.linuxbrew/Cellar/cuda/11.0/include/cuda_runtime.h
-rw-r--r-- 1 aznb aznb 87K Jul  9 10:56 /home/aznb/.linuxbrew/Cellar/cuda/11.0/include/cuda_runtime.h

@leofang Is there a way to print out the exact nvcc command line from the build?

Char-Aznable on Jul 14, 2020