sglang: [Bug] Decode Throughput Inconsistency Between bench_serving and Engine Logs

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

Hi, I encountered an inconsistency in decode throughput reporting. When benchmarking with the bench_serving script, the reported TPOT is much lower than the decode throughput logged by the engine. This gap is significant for small models or high concurrency settings.

Reproduction

start the server

python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-0.5B \
  --trust-remote-code \
  --tp 1 \
  --load-format dummy \
  --port 30000 --host 127.0.0.1

benchmark (seqlens 2048 concurrency 16)

python3 -m sglang.bench_serving \
  --backend sglang \
  --dataset-name random \
  --random-range-ratio 1.0 \
  --random-input-len 2048 \
  --random-output-len 256 \
  --num-prompts 16 \
  --max-concurrency 16 \
  --host 127.0.0.1 \
  --port 30000

Observed Results:

  • The bench_serving script reports a median TPOT of 4.45 ms, equating to a token throughput of 224 $\times$ 16 = 3584 tokens/second.
  • However, the engine logs show a decode throughput of 7026 tokens/second.
Image Image

The gap between these metrics is significant and raises concerns about potential discrepancies in throughput measurement.

Please let me know if you need additional details or logs to assist in troubleshooting.

Environment

Python: 3.10.0 (default, Mar 3 2022, 09:58:08) [GCC 7.5.0]
CUDA available: True
NVIDIA H800 GPU
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.6, V12.6.85
CUDA Driver Version: 535.129.03
PyTorch: 2.5.1+cu124
sglang: 0.4.1.post5
flashinfer: 0.1.6+cu124torch2.4
triton: 3.1.0
transformers: 4.48.0
torchao: 0.7.0
numpy: 1.26.4
aiohttp: 3.11.11
fastapi: 0.115.6
hf_transfer: 0.1.9
huggingface_hub: 0.27.1
interegular: 0.3.3
modelscope: 1.22.1
orjson: 3.10.14
packaging: 24.2
psutil: 6.1.1
pydantic: 2.10.5
multipart: 0.0.20 zmq: 26.2.0 uvicorn: 0.34.0 uvloop: 0.21.0 vllm: 0.6.4.post1 openai: 1.59.7 anthropic: 0.43.0 decord: 0.6.0

About this issue

Most upvoted comments

I think I’ve found the cause. In multiple tests, when the backend is specified as “sglang,” the flush_cache function is called to clear the cache.

If flush_cache is executed successfully, there will be an inconsistency in the Decode Throughput between bench_serving and engine logs, specifically, bench_serving will be lower than engine logs.

If flush_cache fails, the difference in Throughput between the two is within an acceptable range.

I speculate that the time loss caused by the successful execution of flush_cache might be the reason. The Decode Throughput in engine logs is not affected by this, but bench_serving is. I’m still not sure how to fix it. Does anyone have any suggestions?

When the backend is specified as “sglang,” the following code is called to flush the cache,code in here:

# Flush cache
if "sglang" in backend:
    requests.post(base_url + "/flush_cache")

The specific implementation of flush_cache can be found here:

def flush_cache(self):
    """Flush the memory pool and cache."""
    if len(self.waiting_queue) == 0 and (
        self.running_batch is None or len(self.running_batch.reqs) == 0
    ):
        self.tree_cache.reset()
        self.tree_cache_metrics = {"total": 0, "hit": 0}
        if self.grammar_backend:
            self.grammar_backend.reset()
        self.req_to_token_pool.clear()
        self.token_to_kv_pool.clear()

        if not self.spec_algorithm.is_none():
            self.draft_worker.model_runner.req_to_token_pool.clear()
            self.draft_worker.model_runner.token_to_kv_pool.clear()

        self.num_generated_tokens = 0
        self.forward_ct_decode = 0
        self.spec_num_total_accepted_tokens = 0
        self.spec_num_total_forward_ct = 0
        torch.cuda.empty_cache()
        logger.info("Cache flushed successfully!")
        if_success = True
    else:
        logging.warning(
            f"Cache not flushed because there are pending requests. "
            f"#queue-req: {len(self.waiting_queue)}, "
            f"#running-req: {0 if self.running_batch is None else len(self.running_batch.reqs)}"
        )
        if_success = False
    return if_success

@zhaochenyang20 Excuse me, I have three more questions, and I’m wondering if there are any relevant materials or suggestions:

  1. Why does flush_cache lead to a decrease in the throughput of bench_serving, rather than an increase? When testing with random data, there should be no input cache in the GPU. How should I analyze this situation?

  2. If the throughput measured with a failed flush_cache is different from that measured with a successful flush_cache, which one should be considered accurate? Should it force flush_cache to succeed?

  3. Should I modify the bench_serving code to ensure flush_cache succeeds before proceeding with the test? Should I submit a pull request for this?

Thank you!

Sure. I see your point. We will check for this.