vllm: [Bug]: RuntimeError on RTX 5090: "no kernel image is available for execution on the device
Your current environment
Describe the bug
When running
with a NVIDIA RTX 5090 GPU, I encountered the following error:
RuntimeError: CUDA error: no kernel image is available for execution on the device
From the logs, it seems that PyTorch does not support the compute capability of the RTX 5090 (sm_120):
To Reproduce
- Use RTX 5090 GPU
- Install vLLM with Docker or system Python environment
- Launch the vLLM OpenAI API server
- Engine fails to start due to CUDA kernel compatibility issue
Environment
- GPU: NVIDIA GeForce RTX 5090
- CUDA Driver Version: 12.8
- CUDA Toolkit: 12.8.93
- NVIDIA Driver: 570.124.06
- PyTorch Version: 2.x (installed via pip)
- vLLM Version: Latest (from PyPI)
- Python Version: 3.10
- OS: Ubuntu 22.04
Additional Context
It seems that the RTX 5090 uses a new compute capability (sm_120), which is currently not supported in the stable PyTorch build I’m using.
Is there a recommended way to run vLLM with this GPU? Should I:
- Switch to a nightly PyTorch build that supports sm_120?
- Build PyTorch from source with
TORCH_CUDA_ARCH_LIST="12.0"? - Wait for official support from PyTorch?
Any guidance or workaround would be greatly appreciated. Thanks!
How you are installing vllm
pip install -vvv vllm
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 8
- Comments: 28 (1 by maintainers)
good news: RTX 5090 + vllm: works with v0.9.0. Wait for the official docker (or build it yourself)
vllm v0.9.0 has been released 4 days ago!
Why is that big news?
v0.8.5.post1.v0.8.5.post1is a problem because it still uses CUDA 12.4.1 - so no Blackwell support even though it usestorch==2.8.0.dev20250318+cu128v0.9.0now uses CUDA 12.8.1 in the docker image!If you can’t wait for vllm to upload v0.9.0 to Docker Hub, you can already try it out today:
So once they release v0.9.0 to Docker Hub, you will be able to just run vllm from Docker Hub without building anything:
bad news: VLLM still lacks FP8 support on RTX 50xx (SM120)
This does not work for me for fp8 unfortunately:
Same errors as usual for fp8:
edit: errors-details.txt added
System Info:
Are you sure you ran fp8 like that? According to https://github.com/vllm-project/vllm/issues/18153#issuecomment-2888576119
Will try out https://github.com/vllm-project/vllm/issues/14452#issuecomment-2862553056 now, looks promising: https://github.com/vllm-project/vllm/compare/main...kaln27:vllm:main
You have to build the image yourself
here’s the docker that you can try.
clone the vllm repo and place this docker compose right OUTSIDE the repo
To get FP8 models to work on Blackwell
After building the original image, install triton from souce to the image
I have the exact same issue with the RTX 5090. I found no solution so far.
I have built vllm docker for the sm_120 arch which you can try - the docker image is uploaded to the docker hub - voipmonitor/inference-vllm:25.05
in case you want to build it on your machine - here is the Dockerfile I’m using (not all things inside are maybe correct, use on your own risk)
please note that I’m using patched branch with FP8 support on blackwell. If you do not need FP8 you can change the git to the main branch
This will be resolved in the next release! We landed support for SM120 in the wheel and docker image https://github.com/vllm-project/vllm/pull/19794
by the way if anyone builds vLLM based on the current main branch of https://github.com/vllm-project/vllm with
--build-arg torch_cuda_arch_list='12.0'and is wondering why it does not work anymore (in contrast to vLLM builds before 11th June), check out https://github.com/vllm-project/vllm/issues/19810@cyril23 I have tested kaln27 and TensorRT-LLM FP8 model and I’m getting almost exact same performance on two rtx 5090 - I will retest and compare results with 1x 5090 variant to exclude pcie bottleneck. But it seems both vllm and TensorRT-LLM provides nearly same speed.
@bi1101 so for first step (1), I just need create docker-compose.yaml as you suggested. Should I keep only 12.0
torch_cuda_arch_list: "12.0"for faster build time? For second Dockerfile, Do I just need to includes your two lines only, right?Thanks so much bro
Thank you. I have tried your branch and now I’m able to run FP8 model RedHatAI/Qwen3-30B-A3B-FP8-dynamic though I’m not able to run Qwen/Qwen3-30B-A3B-FP8 - have you tried the Qwen3-30B-A3B-FP8 ? It fails with RuntimeError: Currently, block scaled fp8 gemm is not implemented for Blackwell - it seems like torch does not implement this yet? (or what component it might be?
Regarding the tensor parallelism - I have just tested that if I install vllm and pytorch “pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128” and compile vllm directly on the host (ubuntu 24 lts) with Cuda compilation tools, release 12.8, V12.8.93 - it does not crash. I will try to figure out what differs in all the docker images which crashes the parallelism
I’m also curious why when compiling vllm it can use all my 36 cores when nvidia compiler starts while on the docker image it can compile only with 4 parallel tasks (which takes ages) and I have tried almost everything in the docker - injecting -j 36 to the ninja and setting all kind of parameters which should do it - this is annoying when building docker images with vllm
TensorRT-LLM - not yet. I’m new to all of this progressing slowly - but I will try it.
I have also tried Ktransformer with hybrid CPU/GPU configuration and the recent AMX implementation getting interesting results
Link:
https://github.com/triton-lang/triton/issues/6805 https://github.com/pytorch/pytorch/pull/153554
@alew3 sorry, my mistake, the
torch_cuda_arch_list: "7.0 7.5 8.0 8.6 8.9 9.0+PTX 12.0"should not be capitalized, the default arch does not include compute capability 12.0 architecture for 5090, thus the error you are experiencing.Building took ~ 6 hours for me, I recall adding 12.0 architecture took even more time. Adding threads could help, but the build process was bottleneck at some jobs that are not multi threadable, or I’m missing sth. If you found a way to speed up building, pls do let me know.
Also, I experienced some torch compile error with FP8 model so I used
--enforce-eager. After testing, AWQ models offer way better throughput compared to Fp8 so I stick with AWQ, it’'s also easier to Lora finetune AWQ compared to FP8 models imo.