vllm: [Bug]: Engine Core initialization failed. See root cause above
Your current environment
Colab notebooks, A100
🐛 Describe the bug
I have no idea what’s wrong. This model works with normal pipeline, but fails with vllm. It was saved to 16 bit and built off Unsloth/Llama3.18b Instruct
This works -> from transformers import pipeline prompt = ( “Do aliens suffer?” ) gen = pipeline(“text-generation”, model=best_model, tokenizer=tokenizer) print(gen(prompt, max_new_tokens=256, do_sample=True, temperature=0.7)[0][“generated_text”])
This fails-> It does not give me a root cause this error is all I see after it hangs for about 3 minutes
INFO 05-03 23:45:17 [config.py:717] This model supports multiple tasks: {‘generate’, ‘reward’, ‘classify’, ‘score’, ‘embed’}. Defaulting to ‘generate’.
INFO 05-03 23:45:17 [config.py:2003] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 05-03 23:45:19 [utils.py:2382] We must use the spawn multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to ‘spawn’. See https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html#python-multiprocessing for more information. Reason: CUDA is initialized
RuntimeError Traceback (most recent call last) <ipython-input-11-4c7ecb180d12> in <cell line: 0>() 1 from vllm import LLM, SamplingParams ----> 2 llm=LLM(model=“full_model”, tokenizer= “full_model”, gpu_memory_utilization=0.9) 3 prompts = [“Do aliens suffer?”] # Your question 4 5 # Generate responses
8 frames /usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core_client.py in _wait_for_engine_startup(self) 428 if len(events) > 1 or events[0][0] != sync_input_socket: 429 # One of the core processes exited. –> 430 raise RuntimeError("Engine core initialization failed. " 431 “See root cause above.”) 432
RuntimeError: Engine core initialization failed. See root cause above.
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 3
- Comments: 36 (10 by maintainers)
seems to be fixed by import order in my case, importing vllm before transformers seems to fix it. obstruse reasons… may the lord have mercy transformers==4.52.4 vllm==0.8.4
I fixed this problem by adding ‘if name==‘main’:’
No I can’t reformat the logs… This is how they were given…? Do you know the cause of the issue?
Python 多进程问题:如果在使用 Python 多进程时,没有正确设置启动方法,可能会导致此错误。如果日志中出现类似 CUDA was previously initialized 的警告,需要将 vLLM 的使用代码放在 if name == ‘main’: 块中。
You can try https://github.com/vllm-project/vllm/issues/17813#issuecomment-2863718828
Setting CUDA_VISIBLE_DEVICES can also cause the error… When I run my code like
python xxx.pyit does not report the error, but withCUDA_VISIBLE_DEVICES=4,5,6,7 python xxx.pythe error occursCould the log be reformatted for less readability