vllm: [Bug]: RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
I have updated to the latest version and used the “spawn” method,
export VLLM_WORKER_MULTIPROC_METHOD=spawn
but the error still persists. Could you please help me?
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 2
- Comments: 31 (6 by maintainers)
Update Adding
os.environ['VLLM_WORKER_MULTIPROC_METHOD'] = 'spawn'resolved the issue.I am having the same issue from running this example (https://qwen.readthedocs.io/en/latest/deployment/vllm.html#offline-batched-inference).
vLLM version 0.8.2.
Code:
Error:
as pointed by @jeejeelee , the problem is, the users initialized cuda before calling
vllm.LLM(), which is incompatible with the defaultforkapproach. If they want to useexport VLLM_WORKER_MULTIPROC_METHOD=spawn, then they have to move their code underif __name__ == "__main__".the error reported in https://github.com/vllm-project/vllm/issues/8893#issuecomment-2379405305 , is a shutdown error, and the code already finishes. so it can be safely ignored.
Same using torch2.6.0 and vllm 0.8.1
seems like along with
export VLLM_WORKER_MULTIPROC_METHOD=spawn, you need to put your vLLM code inif __name__ == "__main__"section or a function, then it works fine.same here using v1
@youkaichao @DarkLight1337 The same problem. In vllm 0.7.3 and vllm 0.8.1 If in vllm 0.7.1 not this problem And I found sometimes.
self._enum is show 10 on RTX 4090 24GB *1.It should be always 1 not 10.
errorInfo
If add
will get
Then the program freezes.Unable to end.
I have the same problem when using V1.
Thanks for your solution @youkaichao .
My Qwen2-72B-Instruct-GPTQ-Int4 vLLM inference code works fine when it was v0.6.6. But I got the same error after upgrade vllm from v0.6.6 to v0.7.3,
and I fixed it by adding
export VLLM_WORKER_MULTIPROC_METHOD=spawnand removingdevicein calling vllm.LLM()llm = LLM(model="Qwen2-72B-Instruct-GPTQ-Int4", device=device, tensor_parallel_size=2, trust_remote_code=True)But I am wondering what makes the different, could you provide more details of some reference materials?
it works for me, thanks
You can try commenting out or deleting :