vllm: [Bug]: RuntimeError: CUDA error: no kernel image is available for execution on the device

Your current environment

The output of `python collect_env.py`

PyTorch version: 2.3.0 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.29.5 Libc version: glibc-2.35

Python version: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-5.15.0-107-generic-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 11.7.99 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: Tesla V100-PCIE-32GB

Nvidia driver version: 535.171.04 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.8.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.5.0 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

Versions of relevant libraries: [pip3] ctransformers==0.2.27 [pip3] numpy==1.26.4 [pip3] nvidia-nccl-cu12==2.20.5 [pip3] torch==2.3.0 [pip3] torchaudio==2.3.1+cu121 [pip3] torchmetrics==1.1.2 [pip3] torchvision==0.18.0 [pip3] transformers==4.41.2 [pip3] triton==2.3.0 [conda] blas 1.0 mkl
[conda] ctransformers 0.2.27 pypi_0 pypi [conda] ffmpeg 4.3 hf484d3e_0 pytorch [conda] libjpeg-turbo 2.0.0 h9bf148f_0 pytorch [conda] mkl 2023.1.0 h213fc3f_46344
[conda] mkl-fft 1.3.8 pypi_0 pypi [conda] mkl-random 1.2.4 pypi_0 pypi [conda] mkl-service 2.4.0 pypi_0 pypi [conda] mkl_fft 1.3.8 py311h5eee18b_0
[conda] mkl_random 1.2.4 py311hdb19cb5_0
[conda] numpy 1.26.4 pypi_0 pypi [conda] numpy-base 1.26.4 py311hf175353_0
[conda] nvidia-nccl-cu12 2.20.5 pypi_0 pypi [conda] pytorch 2.3.0 py3.11_cuda12.1_cudnn8.9.2_0 pytorch [conda] pytorch-cuda 12.1 ha16c6d3_5 pytorch [conda] pytorch-mutex 1.0 cuda pytorch [conda] torch 2.3.0 pypi_0 pypi [conda] torchaudio 2.3.0 pypi_0 pypi [conda] torchmetrics 1.1.2 pypi_0 pypi [conda] torchtriton 2.3.0 py311 pytorch [conda] torchvision 0.18.0 pypi_0 pypi [conda] transformers 4.41.2 pypi_0 pypi [conda] triton 2.3.0 pypi_0 pypi ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: 0.5.0 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X SYS SYS SYS 0-9,20-29 0 N/A GPU1 SYS X NODE NODE 10-19,30-39 1 N/A GPU2 SYS NODE X NODE 10-19,30-39 1 N/A GPU3 SYS NODE NODE X 10-19,30-39 1 N/A

🐛 Describe the bug

This is the message I get:

[rank0]:     out = torch.empty_like(x)
[rank0]:           ^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: CUDA error: no kernel image is available for execution on the device
[rank0]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

And this is my code for vllm:

llm = LLM(model="yanolja/EEVE-Korean-Instruct-10.8B-v1.0", dtype='half', enable_lora=True)
sampling_params = SamplingParams(
    temperature=0.0,
    max_tokens=100,
    stop=["<|im_end|>"],
)

def test_gen(test_dataset):

    for x in test_dataset:
        question = x["question"]
        answer = x["answer"]
        
        prompts = [
            "Answer the question.\nHuman: {answer}\nAssistant:\n"
        ]

        outputs = llm.generate(
            prompts,
            sampling_params,
            lora_request=LoRARequest("eeve_adapter", 1, hparams.adapter_path),
)

test_gen(datasets['test'])

I tried to reinstall vllm and other libs but I still get this issue. Does anyone know what’s wrong? Please ask for further info if needed. Thanks a lot in advance.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 22 (2 by maintainers)

Most upvoted comments

[rank0]: out = torch.empty_like(x) [rank0]: ^^^^^^^^^^^^^^^^^^^

看起来错误来自 pytorch。你可以尝试:

import torch
a = torch.ones((5,)).cuda()
print(a.sum().item())

i meet the same problem

torch.empty_like(torch.tensor([1])) tensor([139918808922800]) a = torch.ones((5,)).cuda() print(a.sum().item()) 5.0

but [rank0]: out = torch.empty_like(x) [rank0]: RuntimeError: CUDA error: no kernel image is available for execution on the device [rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. [rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1. [rank0]: Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

我的Quadro RTX6000上也有相同的问题