vllm: [Performance]: phi 3.5 vision model consuming high CPU RAM and the process getting killed
Proposal to improve performance
I am trying to run phi3.5 vision instruct model with around 10k prompts. What I noticed with the increase in prompts my CPU RAM consumption keeps increasing and eventually the process gets killed. Its running fine for say small sample like 1000 prompts. My system configuration is 48 GB VRAM and 64GB CPU RAM. Noticed a similar pattern with PIXTRAL-12B-2409. Has anyone faced this issue?
I have tried the implementation by passing in batches of 1000 to llm.generate but still the CPU RAM keeps increasing Below is the code implementation: Ima using two images per prompt from vllm import LLM, SamplingParams llm = LLM( model=“microsoft/Phi-3.5-vision-instruct”, gpu_memory_utilization=0.7, trust_remote_code=True, max_model_len=4096, limit_mm_per_prompt={“image”: 4}, ) sampling_params = SamplingParams(max_tokens=100, temperature=0.0) outputs = llm.generate(prompt_list, sampling_params=sampling_params)
Report of performance regression
No response
Misc discussion on performance
No response
Your current environment (if you think it is necessary)
The output of `python collect_env.py`
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 37 (9 by maintainers)
Thanks, I can reproduce this issue with the provided script. Let me use vLLM’s profiler to see what’s wrong.