sglang: [Bug] Decode Throughput Inconsistency Between bench_serving and Engine Logs
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
Hi, I encountered an inconsistency in decode throughput reporting. When benchmarking with the bench_serving script, the reported TPOT is much lower than the decode throughput logged by the engine. This gap is significant for small models or high concurrency settings.
Reproduction
start the server
python -m sglang.launch_server \
--model-path Qwen/Qwen2.5-0.5B \
--trust-remote-code \
--tp 1 \
--load-format dummy \
--port 30000 --host 127.0.0.1
benchmark (seqlens 2048 concurrency 16)
python3 -m sglang.bench_serving \
--backend sglang \
--dataset-name random \
--random-range-ratio 1.0 \
--random-input-len 2048 \
--random-output-len 256 \
--num-prompts 16 \
--max-concurrency 16 \
--host 127.0.0.1 \
--port 30000
Observed Results:
- The bench_serving script reports a median TPOT of 4.45 ms, equating to a token throughput of 224 $\times$ 16 = 3584 tokens/second.
- However, the engine logs show a decode throughput of 7026 tokens/second.
The gap between these metrics is significant and raises concerns about potential discrepancies in throughput measurement.
Please let me know if you need additional details or logs to assist in troubleshooting.
Environment
Python: 3.10.0 (default, Mar 3 2022, 09:58:08) [GCC 7.5.0]
CUDA available: True
NVIDIA H800 GPU
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.6, V12.6.85
CUDA Driver Version: 535.129.03
PyTorch: 2.5.1+cu124
sglang: 0.4.1.post5
flashinfer: 0.1.6+cu124torch2.4
triton: 3.1.0
transformers: 4.48.0
torchao: 0.7.0
numpy: 1.26.4
aiohttp: 3.11.11
fastapi: 0.115.6
hf_transfer: 0.1.9
huggingface_hub: 0.27.1
interegular: 0.3.3
modelscope: 1.22.1
orjson: 3.10.14
packaging: 24.2
psutil: 6.1.1
pydantic: 2.10.5
multipart: 0.0.20
zmq: 26.2.0
uvicorn: 0.34.0
uvloop: 0.21.0
vllm: 0.6.4.post1
openai: 1.59.7
anthropic: 0.43.0
decord: 0.6.0
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 16
I think I’ve found the cause. In multiple tests, when the backend is specified as “sglang,” the
flush_cachefunction is called to clear the cache.If
flush_cacheis executed successfully, there will be an inconsistency in the Decode Throughput betweenbench_servingandengine logs, specifically,bench_servingwill be lower thanengine logs.If
flush_cachefails, the difference in Throughput between the two is within an acceptable range.I speculate that the time loss caused by the successful execution of
flush_cachemight be the reason. The Decode Throughput inengine logsis not affected by this, butbench_servingis. I’m still not sure how to fix it. Does anyone have any suggestions?When the backend is specified as “sglang,” the following code is called to flush the cache,code in here:
The specific implementation of
flush_cachecan be found here:@zhaochenyang20 Excuse me, I have three more questions, and I’m wondering if there are any relevant materials or suggestions:
Why does
flush_cachelead to a decrease in the throughput ofbench_serving, rather than an increase? When testing with random data, there should be no input cache in the GPU. How should I analyze this situation?If the throughput measured with a failed
flush_cacheis different from that measured with a successfulflush_cache, which one should be considered accurate? Should it forceflush_cacheto succeed?Should I modify the
bench_servingcode to ensureflush_cachesucceeds before proceeding with the test? Should I submit a pull request for this?Thank you!
Sure. I see your point. We will check for this.