vllm: [Bug]: When tensor_parallel_size>1, RuntimeError: Cannot re-initialize CUDA in forked subprocess.
Your current environment
vllm version: ‘0.5.0.post1’
🐛 Describe the bug
When I set tensor_parallel_size=1, it works well. But, if I set tensor_parallel_size>1, below error occurs: RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method. After I add
import torch
import multiprocessing
torch.multiprocessing.set_start_method('spawn')
the same RuntimeError still occurs.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (4 by maintainers)
Commits related to this issue
- fix RuntimeError: Cannot re-initialize CUDA in forked subprocess issue: https://github.com/vllm-project/vllm/issues/6152 — committed to t83714/LIMO by t83714 2 years ago
export VLLM_WORKER_MULTIPROC_METHOD=spawnmay helpIt works for me. Thank you~
it also works for me! thank you!
if you’re using this in a Python notebook run the following first on a reset kernel:
solved problem by add
@rin2401 try to use
distributed_executor_backend="ray"?