vllm: [Bug]: Error in benchmark model with vllm backend for endpoint /v1/chat/completions

Your current environment

The output of `python collect_env.py`
Your output of `python collect_env.py` here

Model Input Dumps

No response

🐛 Describe the bug

vllm benchmark script is falling for endpoint v1/chat/completions

Namespace(backend=‘vllm’, base_url=None, host=‘10.244.2.102’, port=8000, endpoint=‘/v1/chat/completions’, dataset=None, dataset_name=‘sharegpt’, dataset_path=‘ShareGPT_V3_unfiltered_cleaned_split.json’, max_concurrency=None, model=‘Meta-Llama-3.1-70b-instruct’, tokenizer=‘/mnt/models/meta-llama-3-1-70b-instruct/’, best_of=1, use_beam_search=False, num_prompts=1000, logprobs=None, request_rate=3.0, burstiness=1.0, seed=0, trust_remote_code=False, disable_tqdm=True, profile=False, save_result=True, metadata=None, result_dir=‘result/Meta-Llama-3.1-70b-instruct/RR-3-TP-2-PP-1/IL-10’, result_filename=None, ignore_eos=False, percentile_metrics=‘ttft,tpot,itl’, metric_percentiles=‘99’, goodput=None, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, random_input_len=10, random_output_len=512, random_range_ratio=1.0, random_prefix_len=0, hf_subset=None, hf_split=None, hf_output_len=None)

ERROR

Starting initial single prompt test run… Traceback (most recent call last): File “/benchmarking/vllm/benchmarks/benchmark_serving.py”, line 1136, in <module> main(args) File “/benchmarking/vllm/benchmarks/benchmark_serving.py”, line 794, in main benchmark_result = asyncio.run( ^^^^^^^^^^^^ File “/usr/lib/python3.12/asyncio/runners.py”, line 194, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File “/usr/lib/python3.12/asyncio/runners.py”, line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3.12/asyncio/base_events.py”, line 687, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File “/benchmarking/vllm/benchmarks/benchmark_serving.py”, line 489, in benchmark raise ValueError( ValueError: Initial test run failed - Please make sure benchmark arguments are correctly specified. Error: Bad Request

But it works fine for endpoint as ‘v1/completions’

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 23 (7 by maintainers)

Most upvoted comments

I encountered the same problem and found that it was caused by the max_completion_tokens parameter. Modify backend_request_func.py and changed to max_tokens solved this problem.

{"object":"error","message":"[{'type': 'extra_forbidden', 'loc': ('body', 'max_completion_tokens'), 'msg': 'Extra inputs are not permitted', 'input': 119}]","type":"BadRequestError","param":null,"code":400}

I think you need to set --backend openai-chat to use Chat API.