vllm: [Bug] TP=2 fails on dual RTX 5090: TorchInductor compile error or CUDA illegal memory access (TP=1 works)

Your current environment


vLLM Version: 0.9.0
Install method: pip install vllm --extra-index-url https://download.pytorch.org/whl/cu128
PyTorch Version: 3.12.9
CUDA Version: 12.8
Driver Version: 570.86.10
GPU: 2 × RTX 5090
OS: Ubuntu 22.04
</detaisl>

🐛 Describe the bug

my command is:

 python -m vllm.entrypoints.openai.api_server \
  --model NeverSleep/Llama-3-Lumimaid-8B-v0.1 \
  --served-model-name lumimaid \
  --tp 2 \
  --port 48001 \
  --gpu-memory-utilization 0.85 \
  --max_model_len 8192 \
  --enable-prefix-caching \
  --enable-chunked-prefill

log is:

(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]              ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     self.model_runner.profile_run()
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1897, in profile_run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1732, in _dummy_run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     outputs = model(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]               ^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 580, in forward
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 663, in _fn
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     raise e.remove_dynamo_frames() from None  # see TORCHDYNAMO_VERBOSE=1
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1544, in _call_user_compiler
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     raise BackendCompilerFailed(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1519, in _call_user_compiler
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     compiled_fn = compiler_fn(gm, self.example_inputs())
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/repro/after_dynamo.py", line 150, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     compiled_gm = compiler_fn(gm, example_inputs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/repro/after_dynamo.py", line 150, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     compiled_gm = compiler_fn(gm, example_inputs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/__init__.py", line 2392, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return self.compiler_fn(model_, inputs_, **self.kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 498, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     PiecewiseCompileInterpreter(self.split_gm, submod_names_to_compile,
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 273, in run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return super().run(*fake_args)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/interpreter.py", line 171, in run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     self.env[node] = self.run_node(node)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                      ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/interpreter.py", line 240, in run_node
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return getattr(self, n.op)(n.target, args, kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 289, in call_module
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     compiler_manager.compile(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                      ^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 145, in compile
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     compiled_graph, handle = self.compiler.compile(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                              ^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/compiler_interface.py", line 415, in compile
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     assert hash_str is not None, (
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] torch._dynamo.exc.BackendCompilerFailed: backend='<vllm.compilation.backends.VllmBackend object at 0x71d13aff4380>' raised:
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] AssertionError: failed to get the hash of the compiled graph
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]              ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     self.model_runner.profile_run()
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1897, in profile_run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1732, in _dummy_run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     outputs = model(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]               ^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 580, in forward
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 663, in _fn
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     raise e.remove_dynamo_frames() from None  # see TORCHDYNAMO_VERBOSE=1
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1544, in _call_user_compiler
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     raise BackendCompilerFailed(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1519, in _call_user_compiler
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     compiled_fn = compiler_fn(gm, self.example_inputs())
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/repro/after_dynamo.py", line 150, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     compiled_gm = compiler_fn(gm, example_inputs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/repro/after_dynamo.py", line 150, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     compiled_gm = compiler_fn(gm, example_inputs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/__init__.py", line 2392, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return self.compiler_fn(model_, inputs_, **self.kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 498, in __call__
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     PiecewiseCompileInterpreter(self.split_gm, submod_names_to_compile,
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 273, in run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return super().run(*fake_args)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/interpreter.py", line 171, in run
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     self.env[node] = self.run_node(node)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                      ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/fx/interpreter.py", line 240, in run_node
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     return getattr(self, n.op)(n.target, args, kwargs)
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 289, in call_module
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     compiler_manager.compile(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                      ^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/backends.py", line 145, in compile
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     compiled_graph, handle = self.compiler.compile(
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]                              ^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/compiler_interface.py", line 415, in compile
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]     assert hash_str is not None, (
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] torch._dynamo.exc.BackendCompilerFailed: backend='<vllm.compilation.backends.VllmBackend object at 0x71d13aff4380>' raised:
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] AssertionError: failed to get the hash of the compiled graph
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=492198) ERROR 05-27 23:16:12 [multiproc_executor.py:522]
ERROR 05-27 23:16:12 [core.py:500] EngineCore failed to start.
ERROR 05-27 23:16:12 [core.py:500] Traceback (most recent call last):
ERROR 05-27 23:16:12 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 491, in run_engine_core
ERROR 05-27 23:16:12 [core.py:500]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 05-27 23:16:12 [core.py:500]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:16:12 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 390, in __init__
ERROR 05-27 23:16:12 [core.py:500]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 05-27 23:16:12 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 78, in __init__
ERROR 05-27 23:16:12 [core.py:500]     self._initialize_kv_caches(vllm_config)
ERROR 05-27 23:16:12 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 137, in _initialize_kv_caches
ERROR 05-27 23:16:12 [core.py:500]     available_gpu_memory = self.model_executor.determine_available_memory()
ERROR 05-27 23:16:12 [core.py:500]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:16:12 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
ERROR 05-27 23:16:12 [core.py:500]     output = self.collective_rpc("determine_available_memory")
ERROR 05-27 23:16:12 [core.py:500]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:16:12 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc
ERROR 05-27 23:16:12 [core.py:500]     result = get_response(w, dequeue_timeout)
ERROR 05-27 23:16:12 [core.py:500]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:16:12 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 202, in get_response
ERROR 05-27 23:16:12 [core.py:500]     raise RuntimeError(
ERROR 05-27 23:16:12 [core.py:500] RuntimeError: Worker failed with error 'backend='<vllm.compilation.backends.VllmBackend object at 0x71d13aff4380>' raised:
ERROR 05-27 23:16:12 [core.py:500] AssertionError: failed to get the hash of the compiled graph
ERROR 05-27 23:16:12 [core.py:500]
ERROR 05-27 23:16:12 [core.py:500] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
ERROR 05-27 23:16:12 [core.py:500] ', please check the stack trace above for the root cause
(VllmWorker rank=0 pid=492198) Exception ignored in: <function WeakIdKeyDictionary.__init__.<locals>.remove at 0x71d10c5a6980>
(VllmWorker rank=0 pid=492198) Traceback (most recent call last):
(VllmWorker rank=0 pid=492198)   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/weak.py", line 125, in remove
(VllmWorker rank=0 pid=492198)     def remove(k, selfref=ref(self)):
(VllmWorker rank=0 pid=492198)
(VllmWorker rank=0 pid=492198)   File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 450, in signal_handler
(VllmWorker rank=0 pid=492198)     raise SystemExit()
(VllmWorker rank=0 pid=492198) SystemExit:
ERROR 05-27 23:16:13 [multiproc_executor.py:135] Worker proc VllmWorker-1 died unexpectedly, shutting down executor.
Process EngineCore_0:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 504, in run_engine_core
    raise e
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 491, in run_engine_core
    engine_core = EngineCoreProc(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 390, in __init__
    super().__init__(vllm_config, executor_class, log_stats,
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 78, in __init__
    self._initialize_kv_caches(vllm_config)
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 137, in _initialize_kv_caches
    available_gpu_memory = self.model_executor.determine_available_memory()
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
    output = self.collective_rpc("determine_available_memory")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc
    result = get_response(w, dequeue_timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 202, in get_response
    raise RuntimeError(
RuntimeError: Worker failed with error 'backend='<vllm.compilation.backends.VllmBackend object at 0x71d13aff4380>' raised:
AssertionError: failed to get the hash of the compiled graph

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
', please check the stack trace above for the root cause
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 1376, in <module>
    uvloop.run(run_server(args))
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 1324, in run_server
    async with build_async_engine_client(args) as engine_client:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 153, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 185, in build_async_engine_client_from_engine_args
    async_llm = AsyncLLM.from_vllm_config(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/async_llm.py", line 157, in from_vllm_config
    return cls(
           ^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/async_llm.py", line 123, in __init__
    self.engine_core = core_client_class(
                       ^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core_client.py", line 734, in __init__
    super().__init__(
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core_client.py", line 418, in __init__
    self._wait_for_engine_startup(output_address, parallel_config)
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core_client.py", line 484, in _wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "

However: The same setup works on a machine with 2 × RTX 4090. If I set --tp 1, it also works fine on the dual 5090 machine.

I’ve tried setting

export NCCL_P2P_DISABLE=1
export TORCH_COMPILE_DISABLE=1

it shows

(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]              ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     self.model_runner.profile_run()
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1897, in profile_run
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1732, in _dummy_run
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     outputs = model(
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]               ^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 580, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return fn(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 391, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     hidden_states, residual = layer(positions, hidden_states, residual)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 300, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     hidden_states = self.input_layernorm(hidden_states)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/model_executor/custom_op.py", line 23, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return self._forward_method(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/model_executor/layers/layernorm.py", line 145, in forward_native
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     variance = x_var.pow(2).mean(dim=-1, keepdim=True)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] RuntimeError: CUDA error: an illegal memory access was encountered
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]              ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     self.model_runner.profile_run()
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1897, in profile_run
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/v1/worker/gpu_model_runner.py", line 1732, in _dummy_run
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     outputs = model(
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]               ^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 580, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return fn(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 391, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     hidden_states, residual = layer(positions, hidden_states, residual)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/model_executor/models/llama.py", line 300, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     hidden_states = self.input_layernorm(hidden_states)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/model_executor/custom_op.py", line 23, in forward
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     return self._forward_method(*args, **kwargs)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]   File "/home/ubuntu/ml/vllm/vllm/model_executor/layers/layernorm.py", line 145, in forward_native
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]     variance = x_var.pow(2).mean(dim=-1, keepdim=True)
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] RuntimeError: CUDA error: an illegal memory access was encountered
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]
(VllmWorker rank=0 pid=496819) ERROR 05-27 23:44:35 [multiproc_executor.py:522]
ERROR 05-27 23:44:35 [core.py:500] EngineCore failed to start.
ERROR 05-27 23:44:35 [core.py:500] Traceback (most recent call last):
ERROR 05-27 23:44:35 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 491, in run_engine_core
ERROR 05-27 23:44:35 [core.py:500]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 05-27 23:44:35 [core.py:500]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:44:35 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 390, in __init__
ERROR 05-27 23:44:35 [core.py:500]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 05-27 23:44:35 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 78, in __init__
ERROR 05-27 23:44:35 [core.py:500]     self._initialize_kv_caches(vllm_config)
ERROR 05-27 23:44:35 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 137, in _initialize_kv_caches
ERROR 05-27 23:44:35 [core.py:500]     available_gpu_memory = self.model_executor.determine_available_memory()
ERROR 05-27 23:44:35 [core.py:500]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:44:35 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
ERROR 05-27 23:44:35 [core.py:500]     output = self.collective_rpc("determine_available_memory")
ERROR 05-27 23:44:35 [core.py:500]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:44:35 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc
ERROR 05-27 23:44:35 [core.py:500]     result = get_response(w, dequeue_timeout)
ERROR 05-27 23:44:35 [core.py:500]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-27 23:44:35 [core.py:500]   File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 202, in get_response
ERROR 05-27 23:44:35 [core.py:500]     raise RuntimeError(
ERROR 05-27 23:44:35 [core.py:500] RuntimeError: Worker failed with error 'CUDA error: an illegal memory access was encountered
ERROR 05-27 23:44:35 [core.py:500] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
ERROR 05-27 23:44:35 [core.py:500] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
ERROR 05-27 23:44:35 [core.py:500] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
ERROR 05-27 23:44:35 [core.py:500] ', please check the stack trace above for the root cause
ERROR 05-27 23:44:36 [multiproc_executor.py:135] Worker proc VllmWorker-1 died unexpectedly, shutting down executor.
Process EngineCore_0:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 504, in run_engine_core
    raise e
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 491, in run_engine_core
    engine_core = EngineCoreProc(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 390, in __init__
    super().__init__(vllm_config, executor_class, log_stats,
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 78, in __init__
    self._initialize_kv_caches(vllm_config)
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core.py", line 137, in _initialize_kv_caches
    available_gpu_memory = self.model_executor.determine_available_memory()
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
    output = self.collective_rpc("determine_available_memory")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc
    result = get_response(w, dequeue_timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/executor/multiproc_executor.py", line 202, in get_response
    raise RuntimeError(
RuntimeError: Worker failed with error 'CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
', please check the stack trace above for the root cause
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 1376, in <module>
    uvloop.run(run_server(args))
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 1324, in run_server
    async with build_async_engine_client(args) as engine_client:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 153, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/entrypoints/openai/api_server.py", line 185, in build_async_engine_client_from_engine_args
    async_llm = AsyncLLM.from_vllm_config(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/async_llm.py", line 157, in from_vllm_config
    return cls(
           ^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/async_llm.py", line 123, in __init__
    self.engine_core = core_client_class(
                       ^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core_client.py", line 734, in __init__
    super().__init__(
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core_client.py", line 418, in __init__
    self._wait_for_engine_startup(output_address, parallel_config)
  File "/home/ubuntu/ml/vllm/vllm/v1/engine/core_client.py", line 484, in _wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

Most upvoted comments

I fixed the problem for TP>1

python -m pip install -U nvidia-nccl-cu12

The pytorch 2.7.0 dependency installs nccl 2.26.2 and 2.26.5 is needed for multi 5090 support

I fixed the problem for TP>1 python -m pip install -U nvidia-nccl-cu12 The pytorch 2.7.0 dependency installs nccl 2.26.2 and 2.26.5 is needed for multi 5090 support

Can you list the exact process you used to successfully compile and start? I tried installing nccl2.26.5 and still can’t start the server successfully. thank you!

Sure!

git clone https://github.com/vllm-project/vllm
cd vllm
python3.12 -m venv venv
source venv/bin/activate # source venv/bin/activate.fish for fish shell

Install pytorch

python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

python use_existing_torch.py
python -m pip install -r requirements/build.txt
python -m pip install -r requirements/common.txt
python -m pip install -e . --no-build-isolation
python -m pip install -U nvidia-nccl-cu12

Edit: xformers might be needed for some models:

python -m pip install ninja
python -m pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers

VLLM_TEST_FORCE_FP8_MARLIN=1 needed for fp8 models VLLM_FLASH_ATTN_VERSION=2 sometimes needed for flash_attn

VLLM_TEST_FORCE_FP8_MARLIN=1 VLLM_FLASH_ATTN_VERSION=2 CUDA_VISIBLE_DEVICES=2,4 vllm serve /mnt/llms/models/mistralai/Mistral-Nemo-Instruct-2407 -tp 2 --max-model-len 16000 --host 0.0.0.0 --port 5000