vllm: [Bug]: Unable to get vLLM working with RTX 5090

Your current environment

I’ve tried nightly builds, building from source, and docker containers, but I am always seeing the same error:

RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I am able to reproduce the issue by running the following container on Ubuntu 24.04:

docker run --rm \
    --gpus all \
    -v "${HOME}/.cache/huggingface:/root/.cache/huggingface" \
    -p 8000:8000 \
    -e VLLM_USE_FLASH_ATTN=0 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model "${MODEL_NAME}"

Where ${MODEL_NAME} for example could be Qwen/Qwen3-4B-AWQ.

nvidia-smi output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20             Driver Version: 570.133.20     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        Off |   00000000:01:00.0 Off |                  N/A |
|  0%   35C    P8             12W /  575W |      15MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            1553      G   /usr/lib/xorg/Xorg                        4MiB |
+-----------------------------------------------------------------------------------------+

🐛 Describe the bug

I am able to reproduce the bug from the following simple example:

curl -X POST http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
        "model": "Qwen/Qwen3-4B-AWQ",
        "messages": [
          { "role": "system", "content": "You are a helpful assistant." },
          { "role": "user",   "content": "Tell me a joke." }
        ],
        "max_tokens": 50,
        "temperature": 0.7
      }'

FYI ollama works fine as a backend with the 5090, but I want the improved performance and tool calling ability which comes with vllm (OpenAI compatibility). If needed, I can provide the full debug output. I’ve tried with and without the VLLM_USE_FLASH_ATTN environment variable.

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 2
  • Comments: 16 (1 by maintainers)

Most upvoted comments

vllm0.9.1相同的错误

This will be resolved in the next release! We landed support for SM120 in the wheel and docker image https://github.com/vllm-project/vllm/pull/19794