vllm: [Usage]: RuntimeError: NCCL error: invalid usage (run with NCCL_DEBUG=WARN for details)
Your current environment
env torch 2.5.1+cu118 vllm 0.7.2 cuda 12
termial
CUDA_VISIBLE_DEVICES=0,1 VLLM_USE_V1=1 VLLM_WORKER_MULTIPROC_METHOD=spawn vllm serve /home/work/anyujie/Qwen/Qwen2.5-VL-3B-Instruct --gpu-memory-utilization 0.9 --swap-space 16 --max-model-len 4096 --tensor-parallel-size 2 --port 8091 --disable-custom-all-reduce
error
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] WorkerProc hit an exception: %s
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] Traceback (most recent call last):
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/executor/multiproc_executor.py", line 370, in worker_busy_loop
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] output = func(*args, **kwargs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/worker/gpu_worker.py", line 223, in compile_or_warm_up_model
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] self.model_runner.capture_model()
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1036, in capture_model
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] self._dummy_run(num_tokens)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return func(*args, **kwargs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/worker/gpu_model_runner.py", line 894, in _dummy_run
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] hidden_states = model(
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1095, in forward
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] hidden_states = self.language_model.model(
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/compilation/decorators.py", line 245, in __call__
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] model_output = self.forward(*args, **kwargs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/model_executor/models/qwen2.py", line 327, in forward
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] def forward(
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return fn(*args, **kwargs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/fx/graph_module.py", line 784, in call_wrapped
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return self._wrapped_call(self, *args, **kwargs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/fx/graph_module.py", line 361, in __call__
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] raise e
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/fx/graph_module.py", line 348, in __call__
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "<eval_with_key>.74", line 267, in forward
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] submod_2 = self.submod_2(getitem_3, s0, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_parameters_weight_, l_inputs_embeds_, l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight_, l_self_modules_layers_modules_0_modules_mlp_modules_gate_up_proj_parameters_weight_, l_self_modules_layers_modules_0_modules_mlp_modules_down_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_bias_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_, l_positions_, s2); getitem_3 = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_parameters_weight_ = l_inputs_embeds_ = l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight_ = l_self_modules_layers_modules_0_modules_mlp_modules_gate_up_proj_parameters_weight_ = l_self_modules_layers_modules_0_modules_mlp_modules_down_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_bias_ = None
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/compilation/backends.py", line 842, in __call__
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] output = entry.runnable(*args)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/compilation/backends.py", line 239, in compiled_graph
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] graph_output = inductor_compiled_graph(list_args)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 1478, in __call__
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return self.current_callable(inputs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/.cache/vllm/torch_compile_cache/41bb4c971c/rank_1/inductor_cache/jv/cjvnew4xe4km4tkfgn5lbxq3essp3z2p2h2aepjsb27xbv4qr7om.py", line 488, in call
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] buf1 = torch.ops.vllm.all_reduce.default(buf0, 'tp:0')
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/_ops.py", line 716, in __call__
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return self._op(*args, **kwargs)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 111, in all_reduce
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] return group._all_reduce_out_place(tensor)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 370, in _all_reduce_out_place
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] out = pynccl_comm.all_reduce(input_)
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl.py", line 126, in all_reduce
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] self.nccl.ncclAllReduce(buffer_type(in_tensor.data_ptr()),
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 290, in ncclAllReduce
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] self.NCCL_CHECK(self._funcs["ncclAllReduce"](sendbuff, recvbuff, count,
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] raise RuntimeError(f"NCCL error: {error_str}")
(VllmWorker rank=1 pid=12231) ERROR 03-25 21:13:53 multiproc_executor.py:374] RuntimeError: NCCL error: invalid usage (run with NCCL_DEBUG=WARN for details)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] WorkerProc hit an exception: %s
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] Traceback (most recent call last):
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/executor/multiproc_executor.py", line 370, in worker_busy_loop
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/worker/gpu_worker.py", line 223, in compile_or_warm_up_model
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] self.model_runner.capture_model()
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1036, in capture_model
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] self._dummy_run(num_tokens)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return func(*args, **kwargs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/worker/gpu_model_runner.py", line 894, in _dummy_run
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] hidden_states = model(
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1095, in forward
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] hidden_states = self.language_model.model(
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/compilation/decorators.py", line 245, in __call__
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] model_output = self.forward(*args, **kwargs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/model_executor/models/qwen2.py", line 327, in forward
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] def forward(
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return fn(*args, **kwargs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/fx/graph_module.py", line 784, in call_wrapped
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return self._wrapped_call(self, *args, **kwargs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/fx/graph_module.py", line 361, in __call__
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] raise e
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/fx/graph_module.py", line 348, in __call__
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "<eval_with_key>.74", line 267, in forward
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] submod_2 = self.submod_2(getitem_3, s0, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_parameters_weight_, l_inputs_embeds_, l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight_, l_self_modules_layers_modules_0_modules_mlp_modules_gate_up_proj_parameters_weight_, l_self_modules_layers_modules_0_modules_mlp_modules_down_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_bias_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_, l_positions_, s2); getitem_3 = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_parameters_weight_ = l_inputs_embeds_ = l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight_ = l_self_modules_layers_modules_0_modules_mlp_modules_gate_up_proj_parameters_weight_ = l_self_modules_layers_modules_0_modules_mlp_modules_down_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_bias_ = None
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/compilation/backends.py", line 842, in __call__
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] output = entry.runnable(*args)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/compilation/backends.py", line 239, in compiled_graph
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] graph_output = inductor_compiled_graph(list_args)
ERROR 03-25 21:13:53 core.py:210] EngineCore hit an exception: Traceback (most recent call last):
ERROR 03-25 21:13:53 core.py:210] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/engine/core.py", line 202, in run_engine_core
ERROR 03-25 21:13:53 core.py:210] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 03-25 21:13:53 core.py:210] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/engine/core.py", line 156, in __init__
ERROR 03-25 21:13:53 core.py:210] super().__init__(vllm_config, executor_class)
ERROR 03-25 21:13:53 core.py:210] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/engine/core.py", line 54, in __init__
ERROR 03-25 21:13:53 core.py:210] num_gpu_blocks, num_cpu_blocks = self._initialize_kv_caches(
ERROR 03-25 21:13:53 core.py:210] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/engine/core.py", line 88, in _initialize_kv_caches
ERROR 03-25 21:13:53 core.py:210] self.model_executor.initialize(kv_cache_config)
ERROR 03-25 21:13:53 core.py:210] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/executor/abstract.py", line 58, in initialize
ERROR 03-25 21:13:53 core.py:210] self.collective_rpc("compile_or_warm_up_model")
ERROR 03-25 21:13:53 core.py:210] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/executor/multiproc_executor.py", line 133, in collective_rpc
ERROR 03-25 21:13:53 core.py:210] raise e
ERROR 03-25 21:13:53 core.py:210] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/v1/executor/multiproc_executor.py", line 122, in collective_rpc
ERROR 03-25 21:13:53 core.py:210] raise result
ERROR 03-25 21:13:53 core.py:210] RuntimeError: NCCL error: invalid usage (run with NCCL_DEBUG=WARN for details)
ERROR 03-25 21:13:53 core.py:210]
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 1478, in __call__
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return self.current_callable(inputs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/.cache/vllm/torch_compile_cache/41bb4c971c/rank_0/inductor_cache/5z/c5zvmuadomaodz6l6kchh3vaaemvj67qljc2mko5nflojgly6442.py", line 488, in call
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] buf1 = torch.ops.vllm.all_reduce.default(buf0, 'tp:0')
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/torch/_ops.py", line 716, in __call__
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return self._op(*args, **kwargs)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 111, in all_reduce
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] return group._all_reduce_out_place(tensor)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 370, in _all_reduce_out_place
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] out = pynccl_comm.all_reduce(input_)
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl.py", line 126, in all_reduce
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] self.nccl.ncclAllReduce(buffer_type(in_tensor.data_ptr()),
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 290, in ncclAllReduce
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] self.NCCL_CHECK(self._funcs["ncclAllReduce"](sendbuff, recvbuff, count,
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] File "/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] raise RuntimeError(f"NCCL error: {error_str}")
(VllmWorker rank=0 pid=12191) ERROR 03-25 21:13:53 multiproc_executor.py:374] RuntimeError: NCCL error: invalid usage (run with NCCL_DEBUG=WARN for details)
CRITICAL 03-25 21:13:53 core_client.py:158] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
deploy.sh: line 17: 11440 Killed CUDA_VISIBLE_DEVICES=0,1 VLLM_USE_V1=1 VLLM_WORKER_MULTIPROC_METHOD=spawn vllm serve /home/work/anyujie/Qwen/Qwen2.5-VL-3B-Instruct --gpu-memory-utilization 0.9 --swap-space 16 --max-model-len 4096 --tensor-parallel-size 2 --port 8091 --disable-custom-all-reduce
How would you like to use vllm
I want to run inference of a [specific model](put link here). I don’t know how to integrate it with vllm.
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (11 by maintainers)
when i use VLLM_USE_V1=0 it occured: 😭 Process SpawnProcess-1: Traceback (most recent call last): File “/home/work/anyujie/py39/lib/python3.9/multiprocessing/process.py”, line 318, in _bootstrap util._exit_function() File “/home/work/anyujie/py39/lib/python3.9/multiprocessing/util.py”, line 357, in _exit_function p.join() File “/home/work/anyujie/py39/lib/python3.9/multiprocessing/process.py”, line 149, in join res = self._popen.wait(timeout) File “/home/work/anyujie/py39/lib/python3.9/multiprocessing/popen_fork.py”, line 43, in wait return self.poll(os.WNOHANG if timeout == 0.0 else 0) File “/home/work/anyujie/py39/lib/python3.9/multiprocessing/popen_fork.py”, line 27, in poll pid, sts = os.waitpid(self.pid, flag) File “/home/work/anyujie/py39/lib/python3.9/site-packages/vllm/engine/multiprocessing/engine.py”, line 374, in signal_handler raise KeyboardInterrupt(“MQLLMEngine terminated”) KeyboardInterrupt: MQLLMEngine terminated