vllm: [Bug]: Error in benchmark model with vllm backend for endpoint /v1/chat/completions
Your current environment
The output of `python collect_env.py`
Your output of `python collect_env.py` here
Model Input Dumps
No response
🐛 Describe the bug
vllm benchmark script is falling for endpoint v1/chat/completions
Namespace(backend=‘vllm’, base_url=None, host=‘10.244.2.102’, port=8000, endpoint=‘/v1/chat/completions’, dataset=None, dataset_name=‘sharegpt’, dataset_path=‘ShareGPT_V3_unfiltered_cleaned_split.json’, max_concurrency=None, model=‘Meta-Llama-3.1-70b-instruct’, tokenizer=‘/mnt/models/meta-llama-3-1-70b-instruct/’, best_of=1, use_beam_search=False, num_prompts=1000, logprobs=None, request_rate=3.0, burstiness=1.0, seed=0, trust_remote_code=False, disable_tqdm=True, profile=False, save_result=True, metadata=None, result_dir=‘result/Meta-Llama-3.1-70b-instruct/RR-3-TP-2-PP-1/IL-10’, result_filename=None, ignore_eos=False, percentile_metrics=‘ttft,tpot,itl’, metric_percentiles=‘99’, goodput=None, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, random_input_len=10, random_output_len=512, random_range_ratio=1.0, random_prefix_len=0, hf_subset=None, hf_split=None, hf_output_len=None)
ERROR
Starting initial single prompt test run… Traceback (most recent call last): File “/benchmarking/vllm/benchmarks/benchmark_serving.py”, line 1136, in <module> main(args) File “/benchmarking/vllm/benchmarks/benchmark_serving.py”, line 794, in main benchmark_result = asyncio.run( ^^^^^^^^^^^^ File “/usr/lib/python3.12/asyncio/runners.py”, line 194, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File “/usr/lib/python3.12/asyncio/runners.py”, line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3.12/asyncio/base_events.py”, line 687, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File “/benchmarking/vllm/benchmarks/benchmark_serving.py”, line 489, in benchmark raise ValueError( ValueError: Initial test run failed - Please make sure benchmark arguments are correctly specified. Error: Bad Request
But it works fine for endpoint as ‘v1/completions’
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 23 (7 by maintainers)
I encountered the same problem and found that it was caused by the
max_completion_tokensparameter. Modify backend_request_func.py and changed tomax_tokenssolved this problem.I think you need to set
--backend openai-chatto use Chat API.