cupy: AttributeError: 'NoneType' object has no attribute 'record'

CuPy Version : 7.3.0 CUDA Root : /usr/local/cuda CUDA Build Version : 10020 CUDA Driver Version : 10020 CUDA Runtime Version : 10020 cuBLAS Version : 10202 cuFFT Version : 10102 cuRAND Version : 10102 cuSOLVER Version : (10, 3, 0) cuSPARSE Version : 10301 NVRTC Version : (10, 2) cuDNN Build Version : 7605 cuDNN Version : 7605 NCCL Build Version : 2406 NCCL Runtime Version : 2507

If I try to run the following command, I get the following error. cp.asarray(z[cp.newaxis, :, :])

Traceback (most recent call last):
  File "kalman.py", line 125, in <module>
    z = cp.asarray(z[cp.newaxis, :, :])
  File "/home/belt/anaconda3/envs/cusignal/lib/python3.8/site-packages/cupy/creation/from_data.py", line 68, in asarray
    return core.array(a, dtype, False, order)
  File "cupy/core/core.pyx", line 1785, in cupy.core.core.array
  File "cupy/core/core.pyx", line 1862, in cupy.core.core.array
  File "cupy/core/core.pyx", line 1950, in cupy.core.core._send_object_to_gpu
AttributeError: 'NoneType' object has no attribute 'record'

What’s interesting is that it only happens after a run a CuPy kernel. One which doesn’t even access variable z.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 22 (22 by maintainers)

Commits related to this issue

Update stream usage per takagi's suggestions in https://github.com/cupy/cupy/issues/3316 — committed to mnicely/cusignal by mnicely 4 years ago

Most upvoted comments

I mean CuPy’s Stream objects at the following lines:

https://github.com/mnicely/cusignal/blob/a366bb66fb56357d98e77276ba9d031c347febb8/python/cusignal/estimation/filters.py#L150

https://github.com/mnicely/cusignal/blob/a366bb66fb56357d98e77276ba9d031c347febb8/python/cusignal/estimation/filters.py#L160

but, sorry, I misunderstood that these objects are actually referenced by _cupy_predict_wrapper and _cupy_update_wrapper so it’s not the cause.

The actual cause is that, in a previous loop, a KalmanFilter instance’s CuPy Stream is left being used and, in the next loop, it is collected when another KalmanFilter instance is assigned, eventually the CuPy current stream becomes None with a dead weak reference.

I’ve confirmed that with statement for temporary Stream use eliminates the error:

diff --git a/python/cusignal/estimation/_filters.py b/python/cusignal/estimation/_filters.py
index 3686d68..0fbad8b 100644
--- a/python/cusignal/estimation/_filters.py
+++ b/python/cusignal/estimation/_filters.py
@@ -676,8 +676,8 @@ class _cupy_predict_wrapper(object):

         kernel_args = (x.shape[0], alpha_sq, x, F, P, Q)

-        self.stream.use()
-        self.kernel(self.grid, self.block, kernel_args)
+        with self.stream:
+            self.kernel(self.grid, self.block, kernel_args)


 class _cupy_update_wrapper(object):
@@ -696,8 +696,8 @@ class _cupy_update_wrapper(object):

         kernel_args = (x.shape[0], x, z, H, P, R)

-        self.stream.use()
-        self.kernel(self.grid, self.block, kernel_args)
+        with self.stream:
+            self.kernel(self.grid, self.block, kernel_args)


 def _get_backend_kernel(dtype, grid, block, smem, stream, use_numba, k_type):

takagi on Jun 3, 2020

@takagi I believe I made the appropriate changes. Can you try now?

mnicely on May 29, 2020

Thanks @takagi! This is great to understand. It looks like I need to update our library to issue we don’t see this issue again.

mnicely on Jun 3, 2020

I thought I got around but I’m hitting it again. I’m trying to run a loop and it crashes on the first cp.asarray of the second loop (that uses CuPy raw kernels). The Numba kernels have no issues.

See my latest script. kalman_test_script.txt

It this case the error is

Traceback (most recent call last):
  File "../../kalman_test_script.py", line 143, in <module>
    run_test(p, i, n, d)
  File "../../kalman_test_script.py", line 58, in run_test
    cp.asarray(initial_location[cp.newaxis, :, :]), num_points, axis=0
  File "/home/belt/workStuff/rapids/cupy/cupy/creation/from_data.py", line 68, in asarray
    return core.array(a, dtype, False, order)
  File "cupy/core/core.pyx", line 1955, in cupy.core.core.array
  File "cupy/core/core.pyx", line 2032, in cupy.core.core.array
  File "cupy/core/core.pyx", line 2120, in cupy.core.core._send_object_to_gpu
AttributeError: 'NoneType' object has no attribute 'record'

Sorry, I don’t have a better minimal example at the moment.

mnicely on May 11, 2020