vllm: [Bug] TP=2 fails on dual RTX 5090: TorchInductor compile error or CUDA illegal memory access (TP=1 works)
Your current environment
vLLM Version: 0.9.0
Install method: pip install vllm --extra-index-url https://download.pytorch.org/whl/cu128
PyTorch Version: 3.12.9
CUDA Version: 12.8
Driver Version: 570.86.10
GPU: 2 × RTX 5090
OS: Ubuntu 22.04
</detaisl>
🐛 Describe the bug
my command is:
python -m vllm.entrypoints.openai.api_server \
--model NeverSleep/Llama-3-Lumimaid-8B-v0.1 \
--served-model-name lumimaid \
--tp 2 \
--port 48001 \
--gpu-memory-utilization 0.85 \
--max_model_len 8192 \
--enable-prefix-caching \
--enable-chunked-prefill
log is:
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] self.model_runner.profile_run()
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1897, in profile_run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1732, in _dummy_run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] outputs = model(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 580, in forward
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 663, in _fn
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] raise e.remove_dynamo_frames() from None # see TORCHDYNAMO_VERBOSE=1
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1544, in _call_user_compiler
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] raise BackendCompilerFailed(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1519, in _call_user_compiler
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] compiled_fn = compiler_fn(gm, self.example_inputs())
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/repro/after_dynamo.py", line 150, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] compiled_gm = compiler_fn(gm, example_inputs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/repro/after_dynamo.py", line 150, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] compiled_gm = compiler_fn(gm, example_inputs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/__init__.py", line 2392, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return self.compiler_fn(model_, inputs_, **self.kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 498, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] PiecewiseCompileInterpreter(self.split_gm, submod_names_to_compile,
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 273, in run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return super().run(*fake_args)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/interpreter.py", line 171, in run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] self.env[node] = self.run_node(node)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/interpreter.py", line 240, in run_node
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return getattr(self, n.op)(n.target, args, kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 289, in call_module
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] compiler_manager.compile(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 145, in compile
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] compiled_graph, handle = self.compiler.compile(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/compiler_interface.py", line 415, in compile
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] assert hash_str is not None, (
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] torch._dynamo.exc.BackendCompilerFailed: backend='<vllm.compilation.backends.VllmBackend object at 0x71d13aff4380>' raised:
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] AssertionError: failed to get the hash of the compiled graph
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] self.model_runner.profile_run()
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1897, in profile_run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1732, in _dummy_run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] outputs = model(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 580, in forward
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 663, in _fn
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] raise e.remove_dynamo_frames() from None # see TORCHDYNAMO_VERBOSE=1
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1544, in _call_user_compiler
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] raise BackendCompilerFailed(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1519, in _call_user_compiler
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] compiled_fn = compiler_fn(gm, self.example_inputs())
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/repro/after_dynamo.py", line 150, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] compiled_gm = compiler_fn(gm, example_inputs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/repro/after_dynamo.py", line 150, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] compiled_gm = compiler_fn(gm, example_inputs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/__init__.py", line 2392, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return self.compiler_fn(model_, inputs_, **self.kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 498, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] PiecewiseCompileInterpreter(self.split_gm, submod_names_to_compile,
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 273, in run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return super().run(*fake_args)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/interpreter.py", line 171, in run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] self.env[node] = self.run_node(node)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/interpreter.py", line 240, in run_node
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] return getattr(self, n.op)(n.target, args, kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 289, in call_module
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] compiler_manager.compile(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 145, in compile
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] compiled_graph, handle = self.compiler.compile(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/compiler_interface.py", line 415, in compile
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] assert hash_str is not None, (
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] torch._dynamo.exc.BackendCompilerFailed: backend='<vllm.compilation.backends.VllmBackend object at 0x71d13aff4380>' raised:
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] AssertionError: failed to get the hash of the compiled graph
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]
ERROR 05-27 23:16:12 [core.py:500] EngineCore failed to start.
ERROR 05-27 23:16:12 [core.py:500] Traceback (most recent call last):
ERROR 05-27 23:16:12 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 491, in run_engine_core
ERROR 05-27 23:16:12 [core.py:500] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 05-27 23:16:12 [core.py:500] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:16:12 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 390, in __init__
ERROR 05-27 23:16:12 [core.py:500] super().__init__(vllm_config, executor_class, log_stats,
ERROR 05-27 23:16:12 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 78, in __init__
ERROR 05-27 23:16:12 [core.py:500] self._initialize_kv_caches(vllm_config)
ERROR 05-27 23:16:12 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 137, in _initialize_kv_caches
ERROR 05-27 23:16:12 [core.py:500] available_gpu_memory = self.model_executor.determine_available_memory()
ERROR 05-27 23:16:12 [core.py:500] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:16:12 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
ERROR 05-27 23:16:12 [core.py:500] output = self.collective_rpc("determine_available_memory")
ERROR 05-27 23:16:12 [core.py:500] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:16:12 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc
ERROR 05-27 23:16:12 [core.py:500] result = get_response(w, dequeue_timeout)
ERROR 05-27 23:16:12 [core.py:500] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:16:12 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 202, in get_response
ERROR 05-27 23:16:12 [core.py:500] raise RuntimeError(
ERROR 05-27 23:16:12 [core.py:500] RuntimeError: Worker failed with error 'backend='<vllm.compilation.backends.VllmBackend object at 0x71d13aff4380>' raised:
ERROR 05-27 23:16:12 [core.py:500] AssertionError: failed to get the hash of the compiled graph
ERROR 05-27 23:16:12 [core.py:500]
ERROR 05-27 23:16:12 [core.py:500] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
ERROR 05-27 23:16:12 [core.py:500] ', please check the stack trace above for the root cause
(VllmWorker rank=0 pid=492198) Exception ignored in: <function WeakIdKeyDictionary.__init__.<locals>.remove at 0x71d10c5a6980>
(VllmWorker rank=0 pid=492198) Traceback (most recent call last):
(VllmWorker rank=0 pid=492198) File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/weak.py", line 125, in remove
(VllmWorker rank=0 pid=492198) def remove(k, selfref=ref(self)):
(VllmWorker rank=0 pid=492198)
(VllmWorker rank=0 pid=492198) File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 450, in signal_handler
(VllmWorker rank=0 pid=492198) raise SystemExit()
(VllmWorker rank=0 pid=492198) SystemExit:
ERROR 05-27 23:16:13 [multiproc_executor.py:135] Worker proc VllmWorker-1 died unexpectedly, shutting down executor.
Process EngineCore_0:
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 504, in run_engine_core
raise e
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 491, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 390, in __init__
super().__init__(vllm_config, executor_class, log_stats,
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 78, in __init__
self._initialize_kv_caches(vllm_config)
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 137, in _initialize_kv_caches
available_gpu_memory = self.model_executor.determine_available_memory()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
output = self.collective_rpc("determine_available_memory")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc
result = get_response(w, dequeue_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 202, in get_response
raise RuntimeError(
RuntimeError: Worker failed with error 'backend='<vllm.compilation.backends.VllmBackend object at 0x71d13aff4380>' raised:
AssertionError: failed to get the hash of the compiled graph
Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
', please check the stack trace above for the root cause
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 1376, in <module>
uvloop.run(run_server(args))
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 1324, in run_server
async with build_async_engine_client(args) as engine_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 153, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 185, in build_async_engine_client_from_engine_args
async_llm = AsyncLLM.from_vllm_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/engine/async_llm.py", line 157, in from_vllm_config
return cls(
^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/engine/async_llm.py", line 123, in __init__
self.engine_core = core_client_class(
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core_client.py", line 734, in __init__
super().__init__(
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core_client.py", line 418, in __init__
self._wait_for_engine_startup(output_address, parallel_config)
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core_client.py", line 484, in _wait_for_engine_startup
raise RuntimeError("Engine core initialization failed. "
However: The same setup works on a machine with 2 × RTX 4090. If I set --tp 1, it also works fine on the dual 5090 machine.
I’ve tried setting
export NCCL_P2P_DISABLE=1
export TORCH_COMPILE_DISABLE=1
it shows
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] self.model_runner.profile_run()
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1897, in profile_run
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1732, in _dummy_run
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] outputs = model(
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 580, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return fn(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 391, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] hidden_states, residual = layer(positions, hidden_states, residual)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 300, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] hidden_states = self.input_layernorm(hidden_states)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/model_executor/custom_op.py", line 23, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return self._forward_method(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/model_executor/layers/layernorm.py", line 145, in forward_native
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] variance = x_var.pow(2).mean(dim=-1, keepdim=True)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] RuntimeError: CUDA error: an illegal memory access was encountered
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] self.model_runner.profile_run()
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1897, in profile_run
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return func(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1732, in _dummy_run
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] outputs = model(
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 580, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return fn(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 391, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] hidden_states, residual = layer(positions, hidden_states, residual)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 300, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] hidden_states = self.input_layernorm(hidden_states)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/model_executor/custom_op.py", line 23, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] return self._forward_method(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] File "/home/ubuntu/ml/vllm/vllm/model_executor/layers/layernorm.py", line 145, in forward_native
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] variance = x_var.pow(2).mean(dim=-1, keepdim=True)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] RuntimeError: CUDA error: an illegal memory access was encountered
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]
ERROR 05-27 23:44:35 [core.py:500] EngineCore failed to start.
ERROR 05-27 23:44:35 [core.py:500] Traceback (most recent call last):
ERROR 05-27 23:44:35 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 491, in run_engine_core
ERROR 05-27 23:44:35 [core.py:500] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 05-27 23:44:35 [core.py:500] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:44:35 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 390, in __init__
ERROR 05-27 23:44:35 [core.py:500] super().__init__(vllm_config, executor_class, log_stats,
ERROR 05-27 23:44:35 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 78, in __init__
ERROR 05-27 23:44:35 [core.py:500] self._initialize_kv_caches(vllm_config)
ERROR 05-27 23:44:35 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 137, in _initialize_kv_caches
ERROR 05-27 23:44:35 [core.py:500] available_gpu_memory = self.model_executor.determine_available_memory()
ERROR 05-27 23:44:35 [core.py:500] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:44:35 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
ERROR 05-27 23:44:35 [core.py:500] output = self.collective_rpc("determine_available_memory")
ERROR 05-27 23:44:35 [core.py:500] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:44:35 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc
ERROR 05-27 23:44:35 [core.py:500] result = get_response(w, dequeue_timeout)
ERROR 05-27 23:44:35 [core.py:500] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:44:35 [core.py:500] File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 202, in get_response
ERROR 05-27 23:44:35 [core.py:500] raise RuntimeError(
ERROR 05-27 23:44:35 [core.py:500] RuntimeError: Worker failed with error 'CUDA error: an illegal memory access was encountered
ERROR 05-27 23:44:35 [core.py:500] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
ERROR 05-27 23:44:35 [core.py:500] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
ERROR 05-27 23:44:35 [core.py:500] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
ERROR 05-27 23:44:35 [core.py:500] ', please check the stack trace above for the root cause
ERROR 05-27 23:44:36 [multiproc_executor.py:135] Worker proc VllmWorker-1 died unexpectedly, shutting down executor.
Process EngineCore_0:
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 504, in run_engine_core
raise e
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 491, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 390, in __init__
super().__init__(vllm_config, executor_class, log_stats,
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 78, in __init__
self._initialize_kv_caches(vllm_config)
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 137, in _initialize_kv_caches
available_gpu_memory = self.model_executor.determine_available_memory()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
output = self.collective_rpc("determine_available_memory")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc
result = get_response(w, dequeue_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 202, in get_response
raise RuntimeError(
RuntimeError: Worker failed with error 'CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
', please check the stack trace above for the root cause
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 1376, in <module>
uvloop.run(run_server(args))
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 1324, in run_server
async with build_async_engine_client(args) as engine_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 153, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 185, in build_async_engine_client_from_engine_args
async_llm = AsyncLLM.from_vllm_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/engine/async_llm.py", line 157, in from_vllm_config
return cls(
^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/engine/async_llm.py", line 123, in __init__
self.engine_core = core_client_class(
^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core_client.py", line 734, in __init__
super().__init__(
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core_client.py", line 418, in __init__
self._wait_for_engine_startup(output_address, parallel_config)
File "/home/ubuntu/ml/vllm/vllm/v1/engine/core_client.py", line 484, in _wait_for_engine_startup
raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 20
I fixed the problem for TP>1
python -m pip install -U nvidia-nccl-cu12The pytorch 2.7.0 dependency installs nccl 2.26.2 and 2.26.5 is needed for multi 5090 support
Sure!
Install pytorch
Edit: xformers might be needed for some models:
VLLM_TEST_FORCE_FP8_MARLIN=1 needed for fp8 models VLLM_FLASH_ATTN_VERSION=2 sometimes needed for flash_attn
VLLM_TEST_FORCE_FP8_MARLIN=1 VLLM_FLASH_ATTN_VERSION=2 CUDA_VISIBLE_DEVICES=2,4 vllm serve /mnt/llms/models/mistralai/Mistral-Nemo-Instruct-2407 -tp 2 --max-model-len 16000 --host 0.0.0.0 --port 5000