vllm: RuntimeError: CUDA error: no kernel image is available for execution on the device
Error:
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
nvcc -V Copyright © 2005-2022 NVIDIA Corporation Built on Tue_May__3_18:49:52_PDT_2022 Cuda compilation tools, release 11.7, V11.7.64 Build cuda_11.7.r11.7/compiler.31294372_0
conda list: cudatoolkit-dev 11.7.0 cudatoolkit 11.7.0 torch 2.0.1+cu117
nvidia-smi A100 80G NVIDIA-SMI 470.141.03 Driver Version: 470.141.03 CUDA Version: 11.4
how to solve this problem? thanks!
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 33 (6 by maintainers)
I met same issue. The problem is I could run vllm on V100 with cuda 11.3, while can not run on A100 with cuda 12.0. I used exact same codes and docker, except cuda.
Solved, by recompiling and reinstalling the lib when deploying on V100. Previously it was compiled on A100.
@Fr4nk1inCs thanks for the message!
There is not technical limitation to running vllm with CC < 7.0 (see https://github.com/vllm-project/vllm/issues/963#issuecomment-1714100911), apart from not being able to load some models like Falcon. The workaround is to build from source and comment out a couple lines: https://github.com/vllm-project/vllm/issues/463#issuecomment-1636070685
I’ve been able to run vllm without having this issue for weeks. Reverting back to old commits does not resolve it, which probably means that vllm is not responsible for this error. Just curious if anyone else has had this happen to them…
I am also experiencing this using both T4 and V100 GPU’s on Colab
@schnurromafia I don’t think Titan Xp is supported, since its compute capacity is 6.1. One of vLLM’s requirements is GPU with compute capability 7.0 or higher.
I was going through stale issues (no activity in over 3 months) and this one looked to be resolved due to comments such as the one quoted above.
Since vLLM changes so fast, and nobody has encountered this issue in over 3 months, it seemed reasonable to close this issue. If somebody encounters it again, they can open a new issue using the new issue templates making the bug report more actionable.
@pseudotensor are you currently experiencing this issue?
@schnurromafia Thanks for the message! I was also trying to run vLLM on Pascal GPUs. I’ll build it from source and see if it works.
could you check that the problem still exits after rebuilding the repo (
pip install -e .)?