vllm: [Bug]: inter-token latency is lower than TPOT in serving benchmark result

Your current environment

v0.5.2. vLLM env is not an issue so I will just skip the collection process

πŸ› Describe the bug

I am running benchmark tests and notice one potential problem.

Seems the inter-token latency is lower than TPOT. Basically, inter-token latency takes TTFT into the consideration and should be higher than TPOT. However the data shows different result. I have not looked at the code yet and I will try to figure this out

root@fb5250e2ae4c:/workspace# python3 vllm/benchmarks/benchmark_serving.py     --backend vllm     --dataset-name sharegpt     --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json     --model meta-llama/Llama-2-7b-chat-hf     --num-prompts 200     --endpoint /v1/completions     --tokenizer meta-llama/Llama-2-7b-chat-hf     --save-result     2>&1 | tee benchmark_serving.txt
Namespace(backend='vllm', base_url=None, host='localhost', port=8000, endpoint='/v1/completions', dataset=None, dataset_name='sharegpt', dataset_path='./ShareGPT_V3_unfiltered_cleaned_split.json', model='meta-llama/Llama-2-7b-chat-hf', tokenizer='meta-llama/Llama-2-7b-chat-hf', best_of=1, use_beam_search=False, num_prompts=200, sharegpt_output_len=None, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, random_input_len=1024, random_output_len=128, random_range_ratio=1.0, request_rate=inf, seed=0, trust_remote_code=False, disable_tqdm=False, save_result=True, metadata=None, result_dir=None, result_filename=None)
Starting initial single prompt test run...
Initial test run completed. Starting main benchmark run...
Traffic request rate: inf
100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 200/200 [01:12<00:00,  2.74it/s]s
============ Serving Benchmark Result ============
Successful requests:                     200       
Benchmark duration (s):                  72.96     
Total input tokens:                      49490     
Total generated tokens:                  41078     
Request throughput (req/s):              2.74      
Input token throughput (tok/s):          678.34    
Output token throughput (tok/s):         563.04    
---------------Time to First Token----------------
Mean TTFT (ms):                          3594.18   
Median TTFT (ms):                        3685.95   
P99 TTFT (ms):                           7361.98   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          186.90    
Median TPOT (ms):                        121.63    
P99 TPOT (ms):                           966.47    
---------------Inter-token Latency----------------
Mean ITL (ms):                           121.20    
Median ITL (ms):                         92.91     
P99 ITL (ms):                            310.89    
==================================================

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (8 by maintainers)

Most upvoted comments

Thanks @mzmssg, I had also noticed this and have been meaning to open a PR to propose changing the TPOT calculation to a uniform average.

IMO we should do that, and also change the name of what we currently call ITL to something else reflecting that it’s actually time between chunks.

@ywang96

I have noticed a significant difference between TPOT and ITL in my benchmarking. After reviewing the code, I believe the discrepancy primarily arises from how token latency is averaged across different requests.

Suppose we have two requests with e2e latencies $L_1$ and $L_2$(ignoring TTFT), and the number of generated tokens for each request is $N_1$ and $N_2$, respectively. TPOT simply performs an arithmetic mean across different requests:

TPOT = \frac{TPOT_1 + TPOT_2}{2} =  \frac{L_1 / N_1 + L_2 / N_2}{2}

On the other hand, ITL is calculated using a weighted average:

ITL = \frac{L_1 + L_2}{N_1 + N_2}

Naturally, these two methods yield different results. They are equal only when $N_1 = N_2$ or $TPOT_1 = TPOT_2$

I see what you meant about it being captured in the TTFT now – didn’t understand that before but I agree that makes sense. BTW, isn’t it partially captured in the TPOT as well, since you might wait an extra few steps after your final token?

I can throw up a PR to do the name change, and we can discuss further in there. Sounds good?

Actually maybe the variance rather than the mean which would make more sense for this purpose…

which is ITL is reported as smaller than TPOT,

@hyhuang00 yea that’s indeed a good point.

The only possibility I can think of for this is when the model doesn’t generate anything except especial tokens (EOS for example), so the generated text is empty but there’s still ITL record for it. (Here output[i].itl is List[float])

https://github.com/vllm-project/vllm/blob/d9cd78eb718c233ebc5b84377fc2226af7ef0fa2/benchmarks/benchmark_serving.py#L338-L341