vllm: [Bug]: RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

I have updated to the latest version and used the “spawn” method, export VLLM_WORKER_MULTIPROC_METHOD=spawn but the error still persists. Could you please help me?

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 2
  • Comments: 31 (6 by maintainers)

Most upvoted comments

Update Adding os.environ['VLLM_WORKER_MULTIPROC_METHOD'] = 'spawn' resolved the issue.


I am having the same issue from running this example (https://qwen.readthedocs.io/en/latest/deployment/vllm.html#offline-batched-inference).

vLLM version 0.8.2.

Code:

import os
os.environ['HF_HOME'] = 'home/scratch/LLM/cache/'
from huggingface_hub import login
hf_token = "hf_mytokenid"
login(token=hf_token)

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_name = "Qwen/Qwen2.5-7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
llm = LLM(model=model_name)

Error:

INFO 03-29 09:15:54 [config.py:585] This model supports multiple tasks: {'score', 'embed', 'classify', 'reward', 'generate'}. Defaulting to 'generate'.
INFO 03-29 09:15:54 [config.py:1697] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 03-29 09:15:56 [core.py:54] Initializing a V1 LLM engine (v0.8.2) with config: model='Qwen/Qwen2.5-7B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-7B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=Qwen/Qwen2.5-7B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
WARNING 03-29 09:16:14 [utils.py:2321] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x1553f1cbea20>
ERROR 03-29 09:16:14 [core.py:343] EngineCore hit an exception: Traceback (most recent call last):
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 335, in run_engine_core
ERROR 03-29 09:16:14 [core.py:343]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 03-29 09:16:14 [core.py:343]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 290, in __init__
ERROR 03-29 09:16:14 [core.py:343]     super().__init__(vllm_config, executor_class, log_stats)
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 60, in __init__
ERROR 03-29 09:16:14 [core.py:343]     self.model_executor = executor_class(vllm_config)
ERROR 03-29 09:16:14 [core.py:343]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 52, in __init__
ERROR 03-29 09:16:14 [core.py:343]     self._init_executor()
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 46, in _init_executor
ERROR 03-29 09:16:14 [core.py:343]     self.collective_rpc("init_device")
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 03-29 09:16:14 [core.py:343]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 03-29 09:16:14 [core.py:343]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/utils.py", line 2255, in run_method
ERROR 03-29 09:16:14 [core.py:343]     return func(*args, **kwargs)
ERROR 03-29 09:16:14 [core.py:343]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/worker/worker_base.py", line 604, in init_device
ERROR 03-29 09:16:14 [core.py:343]     self.worker.init_device()  # type: ignore
ERROR 03-29 09:16:14 [core.py:343]     ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 103, in init_device
ERROR 03-29 09:16:14 [core.py:343]     torch.cuda.set_device(self.device)
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/torch/cuda/__init__.py", line 476, in set_device
ERROR 03-29 09:16:14 [core.py:343]     torch._C._cuda_setDevice(device)
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
ERROR 03-29 09:16:14 [core.py:343]     raise RuntimeError(
ERROR 03-29 09:16:14 [core.py:343] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
ERROR 03-29 09:16:14 [core.py:343] 
CRITICAL 03-29 09:16:15 [core_client.py:269] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.

You can try commenting out or deleting :

'device = “cuda” if torch.cuda.is_available() else “cpu”

as pointed by @jeejeelee , the problem is, the users initialized cuda before calling vllm.LLM() , which is incompatible with the default fork approach. If they want to use export VLLM_WORKER_MULTIPROC_METHOD=spawn , then they have to move their code under if __name__ == "__main__" .

the error reported in https://github.com/vllm-project/vllm/issues/8893#issuecomment-2379405305 , is a shutdown error, and the code already finishes. so it can be safely ignored.

Same using torch2.6.0 and vllm 0.8.1

seems like along with export VLLM_WORKER_MULTIPROC_METHOD=spawn, you need to put your vLLM code in if __name__ == "__main__" section or a function, then it works fine.

same here using v1

@youkaichao @DarkLight1337 The same problem. In vllm 0.7.3 and vllm 0.8.1 If in vllm 0.7.1 not this problem And I found sometimes.

#/root/anaconda3/lib/python3.12/site-packages/vllm/platforms/interface.py
def is_cuda(self) -> bool:
        return self._enum == PlatformEnum.CUDA

self._enum is show 10 on RTX 4090 24GB *1.It should be always 1 not 10.

errorInfo

WARNING 03-21 17:11:14 [utils.py:2282] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7efba20c0c80>
ERROR 03-21 17:11:14 [core.py:340] EngineCore hit an exception: Traceback (most recent call last):
ERROR 03-21 17:11:14 [core.py:340]   File "/root/anaconda3/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 332, in run_engine_core
ERROR 03-21 17:11:14 [core.py:340]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 03-21 17:11:14 [core.py:340]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-21 17:11:14 [core.py:340]   File "/root/anaconda3/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 287, in __init__
ERROR 03-21 17:11:14 [core.py:340]     super().__init__(vllm_config, executor_class, log_stats)
ERROR 03-21 17:11:14 [core.py:340]   File "/root/anaconda3/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 59, in __init__
ERROR 03-21 17:11:14 [core.py:340]     self.model_executor = executor_class(vllm_config)
ERROR 03-21 17:11:14 [core.py:340]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-21 17:11:14 [core.py:340]   File "/root/anaconda3/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 52, in __init__
ERROR 03-21 17:11:14 [core.py:340]     self._init_executor()
ERROR 03-21 17:11:14 [core.py:340]   File "/root/anaconda3/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 46, in _init_executor
ERROR 03-21 17:11:14 [core.py:340]     self.collective_rpc("init_device")
ERROR 03-21 17:11:14 [core.py:340]   File "/root/anaconda3/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 03-21 17:11:14 [core.py:340]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 03-21 17:11:14 [core.py:340]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-21 17:11:14 [core.py:340]   File "/root/anaconda3/lib/python3.12/site-packages/vllm/utils.py", line 2216, in run_method
ERROR 03-21 17:11:14 [core.py:340]     return func(*args, **kwargs)
ERROR 03-21 17:11:14 [core.py:340]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 03-21 17:11:14 [core.py:340]   File "/root/anaconda3/lib/python3.12/site-packages/vllm/worker/worker_base.py", line 604, in init_device
ERROR 03-21 17:11:14 [core.py:340]     self.worker.init_device()  # type: ignore
ERROR 03-21 17:11:14 [core.py:340]     ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-21 17:11:14 [core.py:340]   File "/root/anaconda3/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 103, in init_device
ERROR 03-21 17:11:14 [core.py:340]     torch.cuda.set_device(self.device)
ERROR 03-21 17:11:14 [core.py:340]   File "/root/anaconda3/lib/python3.12/site-packages/torch/cuda/__init__.py", line 476, in set_device
ERROR 03-21 17:11:14 [core.py:340]     torch._C._cuda_setDevice(device)
ERROR 03-21 17:11:14 [core.py:340]   File "/root/anaconda3/lib/python3.12/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
ERROR 03-21 17:11:14 [core.py:340]     raise RuntimeError(
ERROR 03-21 17:11:14 [core.py:340] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
ERROR 03-21 17:11:14 [core.py:340] 
CRITICAL 03-21 17:11:14 [core_client.py:269] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.

If add

os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"

will get

INFO 03-21 19:05:49 [__init__.py:256] Automatically detected platform cuda.
MODEL_PATH: /mnt/d/Users/Admin/.cache/modelscope/hub/qwen/Qwen2___5-Coder-0___5B-Instruct-AWQ
INFO 03-21 19:07:27 [config.py:583] This model supports multiple tasks: {'embed', 'reward', 'score', 'generate', 'classify'}. Defaulting to 'generate'.
INFO 03-21 19:07:27 [awq_marlin.py:114] The model is convertible to awq_marlin during runtime. Using awq_marlin kernel.
INFO 03-21 19:07:27 [config.py:1693] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 03-21 19:07:27 [cuda.py:95] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 03-21 19:07:40 [__init__.py:256] Automatically detected platform cuda.
MODEL_PATH: /mnt/d/Users/Admin/.cache/modelscope/hub/qwen/Qwen2___5-Coder-0___5B-Instruct-AWQ
INFO 03-21 19:07:50 [config.py:583] This model supports multiple tasks: {'embed', 'reward', 'score', 'generate', 'classify'}. Defaulting to 'generate'.
INFO 03-21 19:07:50 [awq_marlin.py:114] The model is convertible to awq_marlin during runtime. Using awq_marlin kernel.
INFO 03-21 19:07:50 [config.py:1693] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 03-21 19:07:50 [cuda.py:95] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/root/anaconda3/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/multiprocessing/spawn.py", line 131, in _main
    prepare(preparation_data)
  File "/root/anaconda3/lib/python3.12/multiprocessing/spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/root/anaconda3/lib/python3.12/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/runpy.py", line 287, in run_path
    return _run_module_code(code, init_globals, run_name,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/runpy.py", line 98, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/root/anaconda3/lib/python3.12/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/mnt/d/my/work/study/ai/kaggle_code/aimo2/vllm_test.py", line 17, in <module>
    llm = vllm.LLM(
          ^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/site-packages/vllm/utils.py", line 1031, in inner
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 242, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py", line 137, in from_engine_args
    return cls(vllm_config=vllm_config,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py", line 86, in __init__
    self.engine_core = EngineCoreClient.make_client(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 66, in make_client
    return SyncMPClient(vllm_config, executor_class, log_stats)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 337, in __init__
    super().__init__(
  File "/root/anaconda3/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 301, in __init__
    self.resources.proc_handle = BackgroundProcHandle(
                                 ^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/site-packages/vllm/v1/utils.py", line 121, in __init__
    self.proc.start()
  File "/root/anaconda3/lib/python3.12/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/multiprocessing/context.py", line 289, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/root/anaconda3/lib/python3.12/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/root/anaconda3/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/multiprocessing/spawn.py", line 164, in get_preparation_data
    _check_not_importing_main()
  File "/root/anaconda3/lib/python3.12/multiprocessing/spawn.py", line 140, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

        To fix this issue, refer to the "Safe importing of main module"
        section in https://docs.python.org/3/library/multiprocessing.html

Then the program freezes.Unable to end.

I have the same problem when using V1.

You can try commenting out or deleting : 'device = “cuda” if torch.cuda.is_available() else “cpu”

as pointed by @jeejeelee , the problem is, the users initialized cuda before calling vllm.LLM() , which is incompatible with the default fork approach. If they want to use export VLLM_WORKER_MULTIPROC_METHOD=spawn , then they have to move their code under if __name__ == "__main__" .

the error reported in #8893 (comment) , is a shutdown error, and the code already finishes. so it can be safely ignored.

Thanks for your solution @youkaichao .

My Qwen2-72B-Instruct-GPTQ-Int4 vLLM inference code works fine when it was v0.6.6. But I got the same error after upgrade vllm from v0.6.6 to v0.7.3,

and I fixed it by adding export VLLM_WORKER_MULTIPROC_METHOD=spawn and removing device in calling vllm.LLM() llm = LLM(model="Qwen2-72B-Instruct-GPTQ-Int4", device=device, tensor_parallel_size=2, trust_remote_code=True)

But I am wondering what makes the different, could you provide more details of some reference materials?

Update Adding os.environ['VLLM_WORKER_MULTIPROC_METHOD'] = 'spawn' resolved the issue.

I am having the same issue from running this example (https://qwen.readthedocs.io/en/latest/deployment/vllm.html#offline-batched-inference).

vLLM version 0.8.2.

Code:

import os
os.environ['HF_HOME'] = 'home/scratch/LLM/cache/'
from huggingface_hub import login
hf_token = "hf_mytokenid"
login(token=hf_token)

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_name = "Qwen/Qwen2.5-7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
llm = LLM(model=model_name)

Error:

INFO 03-29 09:15:54 [config.py:585] This model supports multiple tasks: {'score', 'embed', 'classify', 'reward', 'generate'}. Defaulting to 'generate'.
INFO 03-29 09:15:54 [config.py:1697] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 03-29 09:15:56 [core.py:54] Initializing a V1 LLM engine (v0.8.2) with config: model='Qwen/Qwen2.5-7B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-7B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=Qwen/Qwen2.5-7B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
WARNING 03-29 09:16:14 [utils.py:2321] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x1553f1cbea20>
ERROR 03-29 09:16:14 [core.py:343] EngineCore hit an exception: Traceback (most recent call last):
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 335, in run_engine_core
ERROR 03-29 09:16:14 [core.py:343]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 03-29 09:16:14 [core.py:343]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 290, in __init__
ERROR 03-29 09:16:14 [core.py:343]     super().__init__(vllm_config, executor_class, log_stats)
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 60, in __init__
ERROR 03-29 09:16:14 [core.py:343]     self.model_executor = executor_class(vllm_config)
ERROR 03-29 09:16:14 [core.py:343]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 52, in __init__
ERROR 03-29 09:16:14 [core.py:343]     self._init_executor()
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 46, in _init_executor
ERROR 03-29 09:16:14 [core.py:343]     self.collective_rpc("init_device")
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 03-29 09:16:14 [core.py:343]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 03-29 09:16:14 [core.py:343]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/utils.py", line 2255, in run_method
ERROR 03-29 09:16:14 [core.py:343]     return func(*args, **kwargs)
ERROR 03-29 09:16:14 [core.py:343]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/worker/worker_base.py", line 604, in init_device
ERROR 03-29 09:16:14 [core.py:343]     self.worker.init_device()  # type: ignore
ERROR 03-29 09:16:14 [core.py:343]     ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 103, in init_device
ERROR 03-29 09:16:14 [core.py:343]     torch.cuda.set_device(self.device)
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/torch/cuda/__init__.py", line 476, in set_device
ERROR 03-29 09:16:14 [core.py:343]     torch._C._cuda_setDevice(device)
ERROR 03-29 09:16:14 [core.py:343]   File "home/.conda/envs/vllm/lib/python3.12/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
ERROR 03-29 09:16:14 [core.py:343]     raise RuntimeError(
ERROR 03-29 09:16:14 [core.py:343] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
ERROR 03-29 09:16:14 [core.py:343] 
CRITICAL 03-29 09:16:15 [core_client.py:269] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.

it works for me, thanks

You can try commenting out or deleting :

 'device = "cuda" if torch.cuda.is_available() else "cpu"