cupy: Printing result from `.max()` errors

Ran the following Python script using CuPy 7.2.0 and got an error. Note the print is what causes the error.

import cupy

a = cupy.arange(5)
r = a.max()
print(r)

Exception:

Traceback (most recent call last):
  File "test_cupy_repr.py", line 5, in <module>
    print(r)
  File "cupy/core/core.pyx", line 1366, in cupy.core.core.ndarray.__str__
  File "cupy/core/core.pyx", line 1456, in cupy.core.core.ndarray.get
  File "cupy/core/core.pyx", line 1462, in cupy.core.core.ndarray.get
  File "cupy/cuda/memory.pyx", line 449, in cupy.cuda.memory.MemoryPointer.copy_to_host
  File "cupy/cuda/runtime.pyx", line 378, in cupy.cuda.runtime.memcpy
  File "cupy/cuda/runtime.pyx", line 201, in cupy.cuda.runtime.check_status
cupy.cuda.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered

Additional exception (during cleanup):

Traceback (most recent call last):
  File "cupy/cuda/driver.pyx", line 247, in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
TypeError: 'NoneType' object is not callable
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy/cuda/driver.pyx", line 247, in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
TypeError: 'NoneType' object is not callable

If it is a bug report, please include the following information:

  • Conditions (you can just paste the output of python -c 'import cupy; cupy.show_config()')
    • CuPy version: 7.2.0
    • OS/Platform: Ubuntu 16.04.5 LTS (Xenial)
    • CUDA version: 10.1.243
    • cuDNN/NCCL version (if applicable): 7.6.5/2.5.7.1
  • Code to reproduce: (see above)
  • Error messages, stack traces, or logs: (see above)

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 34 (27 by maintainers)

Most upvoted comments

It looks like cdef ndarray y was dropped in that commit. Could that be it?

Confirmed this is fixed on conda-forge (we just pushed conda-forge/cupy-feedstock#39).

Ah ok. Yeah I think between that and your testing it looks like PR ( https://github.com/cupy/cupy/pull/3152 ) does indeed fix it then. Glad we are getting some confirmation here. Thanks for doing all of that legwork ๐Ÿ˜„

Thatโ€™s what my doctors tell me ๐Ÿ˜‚ Sure ๐Ÿ™‚

Honestly itโ€™s a wild guess. So it could be totally unrelated ๐Ÿ˜…

Ah ok. Didnโ€™t expect that. Sounds like CUB is the issue then (or something related to it)

Setting cupy.cuda.cub_enabled = False removes the issue! Other reductions have the same behavior as max - failing on the conda build if cub_enabled == True but passing on the conda build if cub_enabled == False, while working on the from-source build.

CUDA toolkit here is 10.0, but a colleague with CUDA 10.2 has seen apparently the same issue.

  1. Could you set cupy.cuda.cub_enabled = False and see if the error persists?
  2. What is your cudatoolkit version (the one installed by conda, not from the system if any)