vllm: [Bug]: When tensor_parallel_size>1, RuntimeError: Cannot re-initialize CUDA in forked subprocess.

Your current environment

vllm version: ‘0.5.0.post1’

🐛 Describe the bug

When I set tensor_parallel_size=1, it works well. But, if I set tensor_parallel_size>1, below error occurs: RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method. After I add

import torch
import multiprocessing
torch.multiprocessing.set_start_method('spawn')

the same RuntimeError still occurs.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (4 by maintainers)

Commits related to this issue

Most upvoted comments

export VLLM_WORKER_MULTIPROC_METHOD=spawn may help

export VLLM_WORKER_MULTIPROC_METHOD=spawn may help

It works for me. Thank you~

export VLLM_WORKER_MULTIPROC_METHOD=spawn may help

it also works for me! thank you!

export VLLM_WORKER_MULTIPROC_METHOD=spawn may help

if you’re using this in a Python notebook run the following first on a reset kernel:

import os
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"

solved problem by add

  1. os.environ[“VLLM_WORKER_MULTIPROC_METHOD”] = “spawn”
  2. if name == “main”:

@rin2401 try to use distributed_executor_backend="ray" ?