vllm: RuntimeError: CUDA error: no kernel image is available for execution on the device
Error:
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
nvcc -V Copyright © 2005-2022 NVIDIA Corporation Built on Tue_May__3_18:49:52_PDT_2022 Cuda compilation tools, release 11.7, V11.7.64 Build cuda_11.7.r11.7/compiler.31294372_0
conda list: cudatoolkit-dev 11.7.0 cudatoolkit 11.7.0 torch 2.0.1+cu117
nvidia-smi A100 80G NVIDIA-SMI 470.141.03 Driver Version: 470.141.03 CUDA Version: 11.4
how to solve this problem? thanks!
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 22 (2 by maintainers)
@Fr4nk1inCs thanks for the message!
There is not technical limitation to running vllm with CC < 7.0 (see https://github.com/vllm-project/vllm/issues/963#issuecomment-1714100911), apart from not being able to load some models like Falcon. The workaround is to build from source and comment out a couple lines: https://github.com/vllm-project/vllm/issues/463#issuecomment-1636070685
I’ve been able to run vllm without having this issue for weeks. Reverting back to old commits does not resolve it, which probably means that vllm is not responsible for this error. Just curious if anyone else has had this happen to them…
I met same issue. The problem is I could run vllm on V100 with cuda 11.3, while can not run on A100 with cuda 12.0. I used exact same codes and docker, except cuda.
@schnurromafia Thanks for the message! I was also trying to run vLLM on Pascal GPUs. I’ll build it from source and see if it works.
Solved, by recompiling and reinstalling the lib when deploying on V100. Previously it was compiled on A100.