vllm: [Bug]: RuntimeError on RTX 5090: "no kernel image is available for execution on the device

Your current environment

Describe the bug

When running

vLLM.log

with a NVIDIA RTX 5090 GPU, I encountered the following error:

RuntimeError: CUDA error: no kernel image is available for execution on the device

From the logs, it seems that PyTorch does not support the compute capability of the RTX 5090 (sm_120):

To Reproduce

  1. Use RTX 5090 GPU
  2. Install vLLM with Docker or system Python environment
  3. Launch the vLLM OpenAI API server
  4. Engine fails to start due to CUDA kernel compatibility issue

Environment

  • GPU: NVIDIA GeForce RTX 5090
  • CUDA Driver Version: 12.8
  • CUDA Toolkit: 12.8.93
  • NVIDIA Driver: 570.124.06
  • PyTorch Version: 2.x (installed via pip)
  • vLLM Version: Latest (from PyPI)
  • Python Version: 3.10
  • OS: Ubuntu 22.04

Additional Context

It seems that the RTX 5090 uses a new compute capability (sm_120), which is currently not supported in the stable PyTorch build I’m using.

Is there a recommended way to run vLLM with this GPU? Should I:

  • Switch to a nightly PyTorch build that supports sm_120?
  • Build PyTorch from source with TORCH_CUDA_ARCH_LIST="12.0"?
  • Wait for official support from PyTorch?

Any guidance or workaround would be greatly appreciated. Thanks!

How you are installing vllm

pip install -vvv vllm

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 8
  • Comments: 28 (1 by maintainers)

Most upvoted comments

good news: RTX 5090 + vllm: works with v0.9.0. Wait for the official docker (or build it yourself)

vllm v0.9.0 has been released 4 days ago!

Why is that big news?

If you can’t wait for vllm to upload v0.9.0 to Docker Hub, you can already try it out today:

# 1. clone official vllm github repo
git clone https://github.com/vllm-project/vllm
cd vllm/
git checkout tags/v0.9.0 -b mybranch

# 2. build
DOCKER_BUILDKIT=1 sudo docker build . --target vllm-openai \
  --tag myvllm --file docker/Dockerfile --build-arg max_jobs=4 \
  --build-arg nvcc_threads=1  \
  --build-arg torch_cuda_arch_list="12.0 12.1" \
  --build-arg RUN_WHEEL_CHECK=false

# 3. run
sudo docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    myvllm --model microsoft/Phi-4-mini-instruct

So once they release v0.9.0 to Docker Hub, you will be able to just run vllm from Docker Hub without building anything:

# 1. run :)
sudo docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    vllm/vllm-openai:latest --model microsoft/Phi-4-mini-instruct

bad news: VLLM still lacks FP8 support on RTX 50xx (SM120)

clone the vllm repo and place this docker compose right OUTSIDE the repo To get FP8 models to work on Blackwell After building the original image, install triton from souce to the image

This does not work for me for fp8 unfortunately:

git clone https://github.com/vllm-project/vllm
cd vllm/
git checkout tags/v0.9.0 -b mybranch
sudo docker build . --target vllm-openai --tag myvllm-all \
  --file docker/Dockerfile --build-arg max_jobs=4 \
  --build-arg nvcc_threads=1  \
  --build-arg torch_cuda_arch_list="7.0 7.5 8.0 8.6 8.9 9.0+PTX 12.0 12.1" 
  --build-arg RUN_WHEEL_CHECK=false
# takes quite a long time: Building 10139.6s (42/42) FINISHED 

nano Dockerfile_all.fp8
# BEGIN Dockerfile_all.fp8
FROM myvllm-all
RUN pip install --no-cache-dir git+https://github.com/triton-lang/triton
# END Dockerfile_all.fp8

sudo docker build   -f Dockerfile_all.fp8   -t myvllm-all-fp8 .

sudo docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface     -p 8000:8000 \
  --env "HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN" \
  --env "TORCH-LOGS=+dynamo"     \
  myvllm-all-fp8 --model RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8     \
  --max-model-len 8k

Same errors as usual for fp8:

INFO 05-18 08:58:46 [default_loader.py:278] Loading weights took 5.17 seconds
INFO 05-18 08:58:46 [gpu_model_runner.py:1447] Model loading took 8.4890 GiB and 6.184214 seconds
INFO 05-18 08:58:51 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/d4491b0504/rank_0_0 for vLLM's torch.compile
INFO 05-18 08:58:51 [backends.py:469] Dynamo bytecode transform time: 4.47 s
Process EngineCore_0:
ERROR 05-18 08:58:52 [core.py:489] EngineCore failed to start.
ERROR 05-18 08:58:52 [core.py:489] Traceback (most recent call last):
ERROR 05-18 08:58:52 [core.py:489]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 480, in run_engine_core
ERROR 05-18 08:58:52 [core.py:489]     engine_core = EngineCoreProc(*args, **kwargs)
[...]
ERROR 05-18 08:58:52 [core.py:489]   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/runtime/triton_heuristics.py", line 1153, in make_launcher
ERROR 05-18 08:58:52 [core.py:489]     "launch_enter_hook": binary.__class__.launch_enter_hook,
ERROR 05-18 08:58:52 [core.py:489]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-18 08:58:52 [core.py:489] torch._inductor.exc.InductorError: AttributeError: type object 'CompiledKernel' has no attribute 'launch_enter_hook'

edit: errors-details.txt added

System Info:

  • x64 (AMD 9800X3D)
  • 96 GB memory
  • NVIDIA RTX 5090, 32 GB VRAM
  • Host:
    • Windows 11 24H2
    • Driver: 576.40 from Mon May 12, 2025
    • Nvidia CUDA Toolkit 12.9
  • WSL2: wsl --version -> WSL-Version: 2.4.13.0, Kernelversion: 5.15.167.4-1, WSLg-Version: 1.0.65, MSRDC-Version: 1.2.5716, Direct3D-Version: 1.611.1-81528511, DXCore-Version: 10.0.26100.1-240331-1435.ge-release, Windows-Version: 10.0.26100.3915
  • Running WSL2 with Ubuntu 24 LTS, freshly installed. The only change to WSL I did is create a file “C:\Users\MYUSER.wslconfig” with content
[wsl2]
memory=85GB

Output of nvidia-smi within Ubuntu 24 WSL:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.55.01              Driver Version: 576.40         CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        On  |   00000000:01:00.0  On |                  N/A |
|  0%   47C    P8             28W /  575W |    1400MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Are you sure you ran fp8 like that? According to https://github.com/vllm-project/vllm/issues/18153#issuecomment-2888576119

SM120(5080, 5090) Blackwell support still needs to be added for GEMMS to vLLM. But on the bright side CUTLASS examples and support has released for 80_blackwell_geforce_sparse_gemm and 79_blackwell_geforce_gemm. We would love some community support here.

Will try out https://github.com/vllm-project/vllm/issues/14452#issuecomment-2862553056 now, looks promising: https://github.com/vllm-project/vllm/compare/main...kaln27:vllm:main

You have to build the image yourself

here’s the docker that you can try.

clone the vllm repo and place this docker compose right OUTSIDE the repo

vllm-Qwen3-30B-A3B-AWQ:
    container_name: vllm-Qwen3-30B-A3B-AWQ
    image: vllm-cu128
    build:
      dockerfile: docker/Dockerfile
      context: vllm
      args:
        max_jobs: 16
        RUN_WHEEL_CHECK: "false"
        nvcc_threads: 16
        torch_cuda_arch_list: "7.0 7.5 8.0 8.6 8.9 9.0+PTX 12.0" # Should not be capitalized
    command: [ "--model", "cognitivecomputations/Qwen3-30B-A3B-AWQ", "--trust-remote-code", "--enable-reasoning", "--reasoning-parser", "deepseek_r1"]
    ports:
      - "8000:8000"
    environment:
      - VLLM_USE_V1=1
      - VLLM_MAX_SIZE_MB=10240
      - VLLM_FLASH_ATTN_VERSION=2
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      # - HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    # runtime: nvidia
    volumes:
      - vllm_cache:/root/.cache
      - model_data:/models
    # ipc: host
    # privileged: true
    # restart: unless-stopped

To get FP8 models to work on Blackwell

After building the original image, install triton from souce to the image

FROM vllm-cu128
# Install Triton from source.
RUN pip install git+https://github.com/triton-lang/triton

I have the exact same issue with the RTX 5090. I found no solution so far.

Hi everyone

Anyone found solution, i’m facing issues running lora + bnb 4 quantization

could this be related to the supported arch on v0.9.1

I have built vllm docker for the sm_120 arch which you can try - the docker image is uploaded to the docker hub - voipmonitor/inference-vllm:25.05

in case you want to build it on your machine - here is the Dockerfile I’m using (not all things inside are maybe correct, use on your own risk)

please note that I’m using patched branch with FP8 support on blackwell. If you do not need FP8 you can change the git to the main branch


# Base image as specified
FROM nvcr.io/nvidia/pytorch:25.05-py3

# --- Build Arguments ---
# Use --build-arg to set this. Examples:
# Hopper (H100/H200): CUDA_ARCHS="9.0"
# Ampere (A100):      CUDA_ARCHS="8.0;8.6;8.9"
# Ada Lovelace (4090): CUDA_ARCHS="8.9" (compiles as Ampere, runs on Ada)
ARG CUDA_ARCHS="12.0"
# Corresponding format for cmake/make (e.g., "90" for Hopper, "86" for A100)
ARG CMAKE_CUDA_ARCHS="120-real"

# Set shell to bash for commands like $(nproc)
SHELL ["/bin/bash", "-c"]

# --- Environment Variables for Compilation ---
# This block corresponds to your 'export' commands.
ENV \
    # Core PyTorch Components
    USE_CUDA=1 \
    USE_CUDNN=1 \
    USE_MKLDNN=1 \
    USE_XNNPACK=1 \
    USE_DISTRIBUTED=1 \
    USE_NCCL=1 \
    USE_SYSTEM_NCCL=1 \
    \
    # Inductor + AOT stack
    USE_INDUCTOR=1 \
    USE_AOT_RUNTIME=1 \
    USE_TRITON=1 \
    USE_FUNCTORCH=1 \
    TORCHINDUCTOR_DEV=1 \
    \
    # Misc Build Flags
    BUILD_CAFFE2=0 \
    BUILD_TEST=0 \
    USE_FLASH_ATTENTION=1 \
    USE_TENSORRT=0 \
    \
    # Build Metadata & Options
    PYTORCH_BUILD_NUMBER=1 \
    MAX_JOBS=$(nproc) \
    FLASHINFER_ENABLE_AOT=1 \
    CCACHE_DIR=/root/.ccache \
    \
    # --- CRITICAL: Set CUDA Architectures from ARG ---
    TORCH_CUDA_ARCH_LIST="${CUDA_ARCHS}" \
    FLASH_ATTN_CUDA_ARCHS="${CMAKE_CUDA_ARCHS}"

# Set working directory for all our builds
WORKDIR /workspace

RUN pip install --upgrade "packaging>=24.2"

# Install essential build dependencies
# git, cmake, and ninja-build are usually in the base image, but we ensure they are present.
RUN apt-get update && apt-get install -y --no-install-recommends \
    git \
    cmake \
    ninja-build \
    ccache \
    vim \
    && rm -rf /var/lib/apt/lists/*

# Install build-time and runtime dependencies in fewer layers
RUN pip install --no-cache-dir \
    triton \
    "numpy<2.3" \
    aiohttp protobuf click rich starlette distro httpx jiter pydantic \
    safetensors uvloop uvicorn python-multipart accelerate msgpack llvmlite

# ------------------------------------------------------------------------------
# 1. Compile and Install PyTorch
# ------------------------------------------------------------------------------
#RUN git clone --recursive https://github.com/pytorch/pytorch.git
#WORKDIR /workspace/pytorch

#RUN pip install --no-cache-dir -r requirements.txt

# Build PyTorch. Using --mount for ccache to speed up rebuilds.
# Note: Changed 'install install' to the standard 'install'.
#RUN --mount=type=cache,target=/root/.ccache \
#    MAX_JOBS=$(nproc) CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) python setup.py --verbose install

# ------------------------------------------------------------------------------
# 2. Compile and Install FlashInfer
# ------------------------------------------------------------------------------
WORKDIR /workspace
RUN git clone --recursive https://github.com/flashinfer-ai/flashinfer.git
WORKDIR /workspace/flashinfer

# Build FlashInfer using CMake and Ninja
#RUN  python3 -m flashinfer.aot --out-dir=/root/.cache/flashinfer --f16-dtype bfloat16  --f8-dtype float8_e4m3fn
#RUN  python3 -m flashinfer.aot --out-dir=/root/.cache/flashinfer
RUN --mount=type=cache,target=/root/.ccache \
    mkdir -p build && \
    cd build && \
    cmake .. -DCMAKE_CUDA_ARCHITECTURES="${CMAKE_CUDA_ARCHS}" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=install -GNinja && \
    ninja -j$(nproc)

# Install FlashInfer via pip in editable mode
RUN pip install -e . --no-build-isolation --no-deps --verbose

# ------------------------------------------------------------------------------
# 3. Compile and Install bitsandbytes
# ------------------------------------------------------------------------------
WORKDIR /workspace
RUN git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git
WORKDIR /workspace/bitsandbytes

# Build bitsandbytes
RUN --mount=type=cache,target=/root/.ccache \
    cmake . -DCOMPUTE_CAPABILITY=${CMAKE_CUDA_ARCHS} -DCOMPUTE_BACKEND=cuda -DCMAKE_BUILD_TYPE=Release && \
    make -j$(nproc)

# Install bitsandbytes via pip in editable mode
RUN --mount=type=cache,target=/root/.ccache \
    pip install -e . --no-build-isolation --no-deps --verbose

WORKDIR /workspace

# Set vLLM-specific environment variables
ENV USE_CUDNN=1 \
    USE_CUSPARSELT=1 \
    NCCL_CUMEM_ENABLE=1

# Clone your specified vLLM fork
RUN git clone https://github.com/cyril23/vllm vllm-kaln27-updated/
WORKDIR /workspace/vllm-kaln27-updated/

# vLLM build process
RUN python use_existing_torch.py
RUN pip install --no-cache-dir -r requirements/build.txt
RUN pip install --no-cache-dir -r requirements/common.txt
RUN pip install --no-cache-dir setuptools_scm

# Build vLLM in development mode. Using --mount for ccache.
RUN --mount=type=cache,target=/root/.ccache \
    #MAX_JOBS=$(nproc) CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) python setup.py --verbose develop
    MAX_JOBS=$(nproc) CMAKE_BUILD_PARALLEL_LEVEL=$(nproc)  pip install . --no-deps --no-build-isolation --verbose

# ------------------------------------------------------------------------------
# Finalization
# ------------------------------------------------------------------------------
# Clean up build directories to reduce final image size
#RUN rm -rf /workspace

# Set the final working directory
WORKDIR /

This will be resolved in the next release! We landed support for SM120 in the wheel and docker image https://github.com/vllm-project/vllm/pull/19794

by the way if anyone builds vLLM based on the current main branch of https://github.com/vllm-project/vllm with --build-arg torch_cuda_arch_list='12.0' and is wondering why it does not work anymore (in contrast to vLLM builds before 11th June), check out https://github.com/vllm-project/vllm/issues/19810

@cyril23 I have tested kaln27 and TensorRT-LLM FP8 model and I’m getting almost exact same performance on two rtx 5090 - I will retest and compare results with 1x 5090 variant to exclude pcie bottleneck. But it seems both vllm and TensorRT-LLM provides nearly same speed.

@techcomthanh You should build 2 images

  1. Just build the dockerfile from source (do not modify it). Remember to add torch_cuda_arch_list 12.0 to the build args (12.0 is the arch of 5090, that is the reason for no kernel image is available, in the lastest docker file, this hasn’t been added, Black well sm100 (B100 B200, etc) might has been added, but not sm120 (5090)
  2. Create a separate Dockerfile for fp8 like I mentioned above to build on top of the vllm-12.8 image, you will want to build this to a different image name like vllm-12.8-triton to avoid overwriting the vllm-12.8 image.

Note: The FP8 worked for Qwen 3 fp8 at the time I posted the comment, but since then I have settled on AWQ so I haven’t touched the FP8, so that may have changed, according to some of the comments above, they reported Fp8 is not yet supported and did not work. You may want to experiment with V0 & V1 and different vllm args to get Fp8 working, I recall I used V0 for Fp8 but I do not remember for sure.

@bi1101 so for first step (1), I just need create docker-compose.yaml as you suggested. Should I keep only 12.0 torch_cuda_arch_list: "12.0" for faster build time? For second Dockerfile, Do I just need to includes your two lines only, right?

Thanks so much bro

@cyril23 have you managed to rum vllm with the FP8 models

@voipmonitor I’ve successfully tested vLLM with FP8 models on a single RTX 5090. I haven’t tried tensor parallelism since I only have one GPU.

Thank you. I have tried your branch and now I’m able to run FP8 model RedHatAI/Qwen3-30B-A3B-FP8-dynamic though I’m not able to run Qwen/Qwen3-30B-A3B-FP8 - have you tried the Qwen3-30B-A3B-FP8 ? It fails with RuntimeError: Currently, block scaled fp8 gemm is not implemented for Blackwell - it seems like torch does not implement this yet? (or what component it might be?

(VllmWorker rank=0 pid=722847) ERROR 06-05 14:28:26 [multiproc_executor.py:522]     torch.ops._C.cutlass_scaled_mm(out, a, b, scale_a, scale_b, bias)
(VllmWorker rank=0 pid=722847) ERROR 06-05 14:28:26 [multiproc_executor.py:522]   File "/root/miniconda3/envs/vllm-clean2/lib/python3.11/site-packages/torch/_ops.py", line 1208, in __call__
(VllmWorker rank=0 pid=722847) ERROR 06-05 14:28:26 [multiproc_executor.py:522]     return self._op(*args, **(kwargs or {}))
(VllmWorker rank=0 pid=722847) ERROR 06-05 14:28:26 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=722847) ERROR 06-05 14:28:26 [multiproc_executor.py:522] RuntimeError: Currently, block scaled fp8 gemm is not implemented for Blackwell

Regarding the tensor parallelism - I have just tested that if I install vllm and pytorch “pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128” and compile vllm directly on the host (ubuntu 24 lts) with Cuda compilation tools, release 12.8, V12.8.93 - it does not crash. I will try to figure out what differs in all the docker images which crashes the parallelism

I’m also curious why when compiling vllm it can use all my 36 cores when nvidia compiler starts while on the docker image it can compile only with 4 parallel tasks (which takes ages) and I have tried almost everything in the docker - injecting -j 36 to the ninja and setting all kind of parameters which should do it - this is annoying when building docker images with vllm

TensorRT-LLM - not yet. I’m new to all of this progressing slowly - but I will try it.

I have also tried Ktransformer with hybrid CPU/GPU configuration and the recent AMX implementation getting interesting results

@alew3 sorry, my mistake, the torch_cuda_arch_list: "7.0 7.5 8.0 8.6 8.9 9.0+PTX 12.0" should not be capitalized, the default arch does not include compute capability 12.0 architecture for 5090, thus the error you are experiencing.

Building took ~ 6 hours for me, I recall adding 12.0 architecture took even more time. Adding threads could help, but the build process was bottleneck at some jobs that are not multi threadable, or I’m missing sth. If you found a way to speed up building, pls do let me know.

Also, I experienced some torch compile error with FP8 model so I used --enforce-eager. After testing, AWQ models offer way better throughput compared to Fp8 so I stick with AWQ, it’'s also easier to Lora finetune AWQ compared to FP8 models imo.