cupy: Printing result from `.max()` errors
Ran the following Python script using CuPy 7.2.0 and got an error. Note the print is what causes the error.
import cupy
a = cupy.arange(5)
r = a.max()
print(r)
Exception:
Traceback (most recent call last):
File "test_cupy_repr.py", line 5, in <module>
print(r)
File "cupy/core/core.pyx", line 1366, in cupy.core.core.ndarray.__str__
File "cupy/core/core.pyx", line 1456, in cupy.core.core.ndarray.get
File "cupy/core/core.pyx", line 1462, in cupy.core.core.ndarray.get
File "cupy/cuda/memory.pyx", line 449, in cupy.cuda.memory.MemoryPointer.copy_to_host
File "cupy/cuda/runtime.pyx", line 378, in cupy.cuda.runtime.memcpy
File "cupy/cuda/runtime.pyx", line 201, in cupy.cuda.runtime.check_status
cupy.cuda.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered
Additional exception (during cleanup):
Traceback (most recent call last):
File "cupy/cuda/driver.pyx", line 247, in cupy.cuda.driver.moduleUnload
File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
TypeError: 'NoneType' object is not callable
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
File "cupy/cuda/driver.pyx", line 247, in cupy.cuda.driver.moduleUnload
File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
TypeError: 'NoneType' object is not callable
If it is a bug report, please include the following information:
- Conditions (you can just paste the output of
python -c 'import cupy; cupy.show_config()')- CuPy version: 7.2.0
- OS/Platform: Ubuntu 16.04.5 LTS (Xenial)
- CUDA version: 10.1.243
- cuDNN/NCCL version (if applicable): 7.6.5/2.5.7.1
- Code to reproduce: (see above)
- Error messages, stack traces, or logs: (see above)
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 34 (27 by maintainers)
It looks like
cdef ndarray ywas dropped in that commit. Could that be it?Confirmed this is fixed on conda-forge (we just pushed conda-forge/cupy-feedstock#39).
Ah ok. Yeah I think between that and your testing it looks like PR ( https://github.com/cupy/cupy/pull/3152 ) does indeed fix it then. Glad we are getting some confirmation here. Thanks for doing all of that legwork ๐
Thatโs what my doctors tell me ๐ Sure ๐
Honestly itโs a wild guess. So it could be totally unrelated ๐
Ah ok. Didnโt expect that. Sounds like CUB is the issue then (or something related to it)
Setting
cupy.cuda.cub_enabled = Falseremoves the issue! Other reductions have the same behavior as max - failing on the conda build if cub_enabled == True but passing on the conda build if cub_enabled == False, while working on the from-source build.CUDA toolkit here is 10.0, but a colleague with CUDA 10.2 has seen apparently the same issue.
cupy.cuda.cub_enabled = Falseand see if the error persists?cudatoolkitversion (the one installed by conda, not from the system if any)