vllm: [Bug]: Running `vllm serve Qwen2.5-VL-72B-Instruct-AWQ` results in an error when upgrading the vLLM version to 0.8.5.
Your current environment
The output of `python collect_env.py`
PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-80GB
Nvidia driver version: 535.161.08
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 14
On-line CPU(s) list: 0-13
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8336C CPU @ 2.30GHz
CPU family: 6
Model: 106
Thread(s) per core: 2
Core(s) per socket: 7
Socket(s): 1
Stepping: 6
BogoMIPS: 4589.21
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves wbnoinvd arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid md_clear arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 336 KiB (7 instances)
L1i cache: 224 KiB (7 instances)
L2 cache: 8.8 MiB (7 instances)
L3 cache: 54 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-13
Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.3.0
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.51.2
[pip3] triton==3.2.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.8.5
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 0-13 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY
🐛 Describe the bug
Reproduce command: vllm serve /data/modelscope/Qwen2.5-VL-72B-Instruct-AWQ --max-model-len=32768 --limit-mm-per-prompt ‘{“image”=16}’ --gpu-memory-utilization=0.9
Error output like below:
ERROR 04-29 10:42:00 [core.py:396] EngineCore failed to start.
ERROR 04-29 10:42:00 [core.py:396] Traceback (most recent call last):
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 387, in run_engine_core
ERROR 04-29 10:42:00 [core.py:396] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 329, in __init__
ERROR 04-29 10:42:00 [core.py:396] super().__init__(vllm_config, executor_class, log_stats,
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 71, in __init__
ERROR 04-29 10:42:00 [core.py:396] self._initialize_kv_caches(vllm_config)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 129, in _initialize_kv_caches
ERROR 04-29 10:42:00 [core.py:396] available_gpu_memory = self.model_executor.determine_available_memory()
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
ERROR 04-29 10:42:00 [core.py:396] output = self.collective_rpc("determine_available_memory")
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 04-29 10:42:00 [core.py:396] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/utils.py", line 2456, in run_method
ERROR 04-29 10:42:00 [core.py:396] return func(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 04-29 10:42:00 [core.py:396] return func(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 183, in determine_available_memory
ERROR 04-29 10:42:00 [core.py:396] self.model_runner.profile_run()
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1651, in profile_run
ERROR 04-29 10:42:00 [core.py:396] hidden_states = self._dummy_run(self.max_num_tokens)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 04-29 10:42:00 [core.py:396] return func(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1497, in _dummy_run
ERROR 04-29 10:42:00 [core.py:396] outputs = model(
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
ERROR 04-29 10:42:00 [core.py:396] return self._call_impl(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
ERROR 04-29 10:42:00 [core.py:396] return forward_call(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1106, in forward
ERROR 04-29 10:42:00 [core.py:396] hidden_states = self.language_model.model(
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 238, in __call__
ERROR 04-29 10:42:00 [core.py:396] output = self.compiled_callable(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 574, in _fn
ERROR 04-29 10:42:00 [core.py:396] return fn(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 325, in forward
ERROR 04-29 10:42:00 [core.py:396] def forward(
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
ERROR 04-29 10:42:00 [core.py:396] return self._call_impl(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
ERROR 04-29 10:42:00 [core.py:396] return forward_call(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn
ERROR 04-29 10:42:00 [core.py:396] return fn(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/fx/graph_module.py", line 822, in call_wrapped
ERROR 04-29 10:42:00 [core.py:396] return self._wrapped_call(self, *args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/fx/graph_module.py", line 400, in __call__
ERROR 04-29 10:42:00 [core.py:396] raise e
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/fx/graph_module.py", line 387, in __call__
ERROR 04-29 10:42:00 [core.py:396] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
ERROR 04-29 10:42:00 [core.py:396] return self._call_impl(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
ERROR 04-29 10:42:00 [core.py:396] return forward_call(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "<eval_with_key>.162", line 2169, in forward
ERROR 04-29 10:42:00 [core.py:396] submod_0 = self.submod_0(l_inputs_embeds_, s0, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_qweight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_scales_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_qzeros_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_g_idx_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_g_idx_sort_indices_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_workspace, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_bias_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_, l_positions_, s2); l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_qweight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_scales_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_qzeros_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_g_idx_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_g_idx_sort_indices_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_workspace = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_bias_ = None
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/compilation/backends.py", line 612, in __call__
ERROR 04-29 10:42:00 [core.py:396] return self.compiled_graph_for_general_shape(*args)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn
ERROR 04-29 10:42:00 [core.py:396] return fn(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1184, in forward
ERROR 04-29 10:42:00 [core.py:396] return compiled_fn(full_args)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 323, in runtime_wrapper
ERROR 04-29 10:42:00 [core.py:396] all_outs = call_func_at_runtime_with_args(
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
ERROR 04-29 10:42:00 [core.py:396] out = normalize_as_list(f(args))
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 672, in inner_fn
ERROR 04-29 10:42:00 [core.py:396] outs = compiled_fn(args)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 490, in wrapper
ERROR 04-29 10:42:00 [core.py:396] return compiled_fn(runtime_args)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 466, in __call__
ERROR 04-29 10:42:00 [core.py:396] return self.current_callable(inputs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/utils.py", line 2128, in run
ERROR 04-29 10:42:00 [core.py:396] return model(new_inputs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/.cache/vllm/torch_compile_cache/ad24c14253/rank_0_0/inductor_cache/qo/cqofcthiyqgq2qgfhymaumxrr5ycw2u6xxcpo74ztvqslexfiejb.py", line 671, in call
ERROR 04-29 10:42:00 [core.py:396] triton_poi_fused_add_4.run(buf12, arg9_1, triton_poi_fused_add_4_xnumel, grid=grid(triton_poi_fused_add_4_xnumel), stream=stream0)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 1034, in run
ERROR 04-29 10:42:00 [core.py:396] self.autotune_to_one_config(*args, grid=grid, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 911, in autotune_to_one_config
ERROR 04-29 10:42:00 [core.py:396] timings = self.benchmark_all_configs(*args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 885, in benchmark_all_configs
ERROR 04-29 10:42:00 [core.py:396] timings = {
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 886, in <dictcomp>
ERROR 04-29 10:42:00 [core.py:396] launcher: self.bench(launcher, *args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 787, in bench
ERROR 04-29 10:42:00 [core.py:396] return benchmarker.benchmark_gpu(kernel_call, rep=40)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/benchmarking.py", line 66, in wrapper
ERROR 04-29 10:42:00 [core.py:396] return fn(self, *args, **kwargs)
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/benchmarking.py", line 202, in benchmark_gpu
ERROR 04-29 10:42:00 [core.py:396] return self.triton_do_bench(_callable, **kwargs, return_mode="median")
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/triton/testing.py", line 118, in do_bench
ERROR 04-29 10:42:00 [core.py:396] di.synchronize()
ERROR 04-29 10:42:00 [core.py:396] File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/cuda/__init__.py", line 985, in synchronize
ERROR 04-29 10:42:00 [core.py:396] return torch._C._cuda_synchronize()
ERROR 04-29 10:42:00 [core.py:396] RuntimeError: CUDA error: an illegal memory access was encountered
ERROR 04-29 10:42:00 [core.py:396] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
ERROR 04-29 10:42:00 [core.py:396] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
ERROR 04-29 10:42:00 [core.py:396] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
ERROR 04-29 10:42:00 [core.py:396]
Process EngineCore_0:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 400, in run_engine_core
raise e
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 387, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 329, in __init__
super().__init__(vllm_config, executor_class, log_stats,
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 71, in __init__
self._initialize_kv_caches(vllm_config)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 129, in _initialize_kv_caches
available_gpu_memory = self.model_executor.determine_available_memory()
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
output = self.collective_rpc("determine_available_memory")
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/utils.py", line 2456, in run_method
return func(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 183, in determine_available_memory
self.model_runner.profile_run()
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1651, in profile_run
hidden_states = self._dummy_run(self.max_num_tokens)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1497, in _dummy_run
outputs = model(
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1106, in forward
hidden_states = self.language_model.model(
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 238, in __call__
output = self.compiled_callable(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 574, in _fn
return fn(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 325, in forward
def forward(
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn
return fn(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/fx/graph_module.py", line 822, in call_wrapped
return self._wrapped_call(self, *args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/fx/graph_module.py", line 400, in __call__
raise e
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/fx/graph_module.py", line 387, in __call__
return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "<eval_with_key>.162", line 2169, in forward
submod_0 = self.submod_0(l_inputs_embeds_, s0, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_qweight_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_scales_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_qzeros_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_g_idx_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_g_idx_sort_indices_, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_workspace, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_bias_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_, l_positions_, s2); l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_qweight_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_scales_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_qzeros_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_g_idx_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_g_idx_sort_indices_ = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_workspace = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_bias_ = None
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/compilation/backends.py", line 612, in __call__
return self.compiled_graph_for_general_shape(*args)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn
return fn(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1184, in forward
return compiled_fn(full_args)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 323, in runtime_wrapper
all_outs = call_func_at_runtime_with_args(
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
out = normalize_as_list(f(args))
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 672, in inner_fn
outs = compiled_fn(args)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 490, in wrapper
return compiled_fn(runtime_args)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 466, in __call__
return self.current_callable(inputs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/utils.py", line 2128, in run
return model(new_inputs)
File "/home/lucas/.cache/vllm/torch_compile_cache/ad24c14253/rank_0_0/inductor_cache/qo/cqofcthiyqgq2qgfhymaumxrr5ycw2u6xxcpo74ztvqslexfiejb.py", line 671, in call
triton_poi_fused_add_4.run(buf12, arg9_1, triton_poi_fused_add_4_xnumel, grid=grid(triton_poi_fused_add_4_xnumel), stream=stream0)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 1034, in run
self.autotune_to_one_config(*args, grid=grid, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 911, in autotune_to_one_config
timings = self.benchmark_all_configs(*args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 885, in benchmark_all_configs
timings = {
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 886, in <dictcomp>
launcher: self.bench(launcher, *args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 787, in bench
return benchmarker.benchmark_gpu(kernel_call, rep=40)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/benchmarking.py", line 66, in wrapper
return fn(self, *args, **kwargs)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/_inductor/runtime/benchmarking.py", line 202, in benchmark_gpu
return self.triton_do_bench(_callable, **kwargs, return_mode="median")
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/triton/testing.py", line 118, in do_bench
di.synchronize()
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/torch/cuda/__init__.py", line 985, in synchronize
return torch._C._cuda_synchronize()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
[rank0]:[W429 10:42:01.769171143 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Traceback (most recent call last):
File "/home/lucas/envs/nlp-vllm/bin/vllm", line 8, in <module>
sys.exit(main())
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/entrypoints/cli/main.py", line 53, in main
args.dispatch_function(args)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd
uvloop.run(run_server(args))
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
return await main
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server
async with build_async_engine_client(args) as engine_client:
File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 178, in build_async_engine_client_from_engine_args
async_llm = AsyncLLM.from_vllm_config(
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 150, in from_vllm_config
return cls(
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 118, in __init__
self.engine_core = core_client_class(
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 642, in __init__
super().__init__(
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 398, in __init__
self._wait_for_engine_startup()
File "/home/lucas/envs/nlp-vllm/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 430, in _wait_for_engine_startup
raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above.
Tips: Everything works fine with vLLM version 0.8.4.
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (2 by maintainers)
When I build from source, it seems ok!