vllm: [Bug]: RuntimeError: CUDA error: no kernel image is available for execution on the device
Your current environment
The output of `python collect_env.py`
PyTorch version: 2.3.0 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.4 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.29.5 Libc version: glibc-2.35
Python version: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-5.15.0-107-generic-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 11.7.99 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: Tesla V100-PCIE-32GB
Nvidia driver version: 535.171.04 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.8.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.5.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.5.0 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True
Versions of relevant libraries:
[pip3] ctransformers==0.2.27
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] torch==2.3.0
[pip3] torchaudio==2.3.1+cu121
[pip3] torchmetrics==1.1.2
[pip3] torchvision==0.18.0
[pip3] transformers==4.41.2
[pip3] triton==2.3.0
[conda] blas 1.0 mkl
[conda] ctransformers 0.2.27 pypi_0 pypi
[conda] ffmpeg 4.3 hf484d3e_0 pytorch
[conda] libjpeg-turbo 2.0.0 h9bf148f_0 pytorch
[conda] mkl 2023.1.0 h213fc3f_46344
[conda] mkl-fft 1.3.8 pypi_0 pypi
[conda] mkl-random 1.2.4 pypi_0 pypi
[conda] mkl-service 2.4.0 pypi_0 pypi
[conda] mkl_fft 1.3.8 py311h5eee18b_0
[conda] mkl_random 1.2.4 py311hdb19cb5_0
[conda] numpy 1.26.4 pypi_0 pypi
[conda] numpy-base 1.26.4 py311hf175353_0
[conda] nvidia-nccl-cu12 2.20.5 pypi_0 pypi
[conda] pytorch 2.3.0 py3.11_cuda12.1_cudnn8.9.2_0 pytorch
[conda] pytorch-cuda 12.1 ha16c6d3_5 pytorch
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torch 2.3.0 pypi_0 pypi
[conda] torchaudio 2.3.0 pypi_0 pypi
[conda] torchmetrics 1.1.2 pypi_0 pypi
[conda] torchtriton 2.3.0 py311 pytorch
[conda] torchvision 0.18.0 pypi_0 pypi
[conda] transformers 4.41.2 pypi_0 pypi
[conda] triton 2.3.0 pypi_0 pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.0
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X SYS SYS SYS 0-9,20-29 0 N/A
GPU1 SYS X NODE NODE 10-19,30-39 1 N/A
GPU2 SYS NODE X NODE 10-19,30-39 1 N/A
GPU3 SYS NODE NODE X 10-19,30-39 1 N/A
🐛 Describe the bug
This is the message I get:
[rank0]: out = torch.empty_like(x)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: CUDA error: no kernel image is available for execution on the device
[rank0]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
And this is my code for vllm:
llm = LLM(model="yanolja/EEVE-Korean-Instruct-10.8B-v1.0", dtype='half', enable_lora=True)
sampling_params = SamplingParams(
temperature=0.0,
max_tokens=100,
stop=["<|im_end|>"],
)
def test_gen(test_dataset):
for x in test_dataset:
question = x["question"]
answer = x["answer"]
prompts = [
"Answer the question.\nHuman: {answer}\nAssistant:\n"
]
outputs = llm.generate(
prompts,
sampling_params,
lora_request=LoRARequest("eeve_adapter", 1, hparams.adapter_path),
)
test_gen(datasets['test'])
I tried to reinstall vllm and other libs but I still get this issue. Does anyone know what’s wrong? Please ask for further info if needed. Thanks a lot in advance.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 22 (2 by maintainers)
i meet the same problem
but [rank0]: out = torch.empty_like(x) [rank0]: RuntimeError: CUDA error: no kernel image is available for execution on the device [rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. [rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1. [rank0]: Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.我的Quadro RTX6000上也有相同的问题