mamba: RuntimeError: CUDA error: no kernel image is available for execution on the device
is a p5200 enough for this?
]
Traceback (most recent call last):
File “/home/user/mamba/simplermambassm.py”, line 259, in <module>
losses = estimate_loss()
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/utils/_contextlib.py”, line 115, in decorate_context
return func(*args, **kwargs)
File “/home/user/mamba/simplermambassm.py”, line 94, in estimate_loss
logits, loss = model(X,Y)
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1527, in _call_impl
return forward_call(*args, **kwargs)
File “/home/user/mamba/simplermambassm.py”, line 210, in forward
x = self.blocks(x) # (B,T,C_e)
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1527, in _call_impl
return forward_call(*args, **kwargs)
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/nn/modules/container.py”, line 215, in forward
input = module(input)
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1527, in _call_impl
return forward_call(*args, **kwargs)
File “/home/user/mamba/simplermambassm.py”, line 188, in forward
x = x + self.sa_head(self.ln1(x))
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1527, in _call_impl
return forward_call(*args, **kwargs)
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/mamba_ssm/modules/mamba_simple.py”, line 149, in forward
out = mamba_inner_fn(
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/mamba_ssm/ops/selective_scan_interface.py”, line 306, in mamba_inner_fn
return MambaInnerFn.apply(xz, conv1d_weight, conv1d_bias, x_proj_weight, delta_proj_weight,
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/autograd/function.py”, line 539, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py”, line 113, in decorate_fwd
return fwd(*args, **kwargs)
File “/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/mamba_ssm/ops/selective_scan_interface.py”, line 181, in forward
conv1d_out = causal_conv1d_cuda.causal_conv1d_fwd(x, conv1d_weight, conv1d_bias, True)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 15
@tridao (I am not sure if this is just a hack, but for us old guys with CCC < 7, can we do this?)
I see that the Quadro P5200 has Cuda Compute capability 6.1. I saw the same error with my GeForce GTX 1070 (which is also Compute Capability 6.1)
I was able to fix it by compiling the causal-conv1d dependency from source, as follows:
Here is where you need to add those lines.
Then, compile it from source with:
CAUSAL_CONV1D_FORCE_BUILD=TRUE pip install .You can use the following script to test whether it is working properly:
EDIT: Just realized the Mamba repo also assumes CCC >= 7. So, I did a similar edit to the mamba setup.py and compiled it with:
henry@henry-gs65:mamba$ MAMBA_FORCE_BUILD=TRUE pip install .(This takes about 10 minutes to compile)
Once doing this, the top-level Mamba demo works:
oops, sorry but I forgot a crucial thing. Mamba states that it requires causal_conv1d version <= 1.0.2. I forgot to mention this. So, you need to do a
git checkout v1.0.2before you do the pip install. From where you are now, I’d say it would be:At this point, it may work 😉 Since Mamba dynamically loads the causal-conv1d python module, no re-compilation of mamba is necessary. But I am not positive of that.
despite installing python3-packaging and pip install packaging (and can confirm I can import packaging)
yay, that did it back in the game =D
Sorry I’m traveling this week but will have time to look into this next week.
close