sglang: [Bug] PD disaggregation, KV transfer slow down under high concurrency

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

When running bench_serving, we send 200 prompts(isl=4k, osl=2k) in one shot, the first 100 reqs’s KV were transferred in 30s. But the second 100 reqs’s KV transfer took 5min.

I’ve print out the prealloc queue and transfer queue size in decode node. We can see the transfer-queue size was decreasing very slowly after 2025-04-16 12:40:10. log.txt

Reproduction

Node1:

MOONCAKE_CONFIG_PATH=./mooncake.json \
  python -m sglang.launch_server \
  --model-path my_model_path \
  --disaggregation-mode prefill \
  --port 30000 --host 192.168.1.1 \
  --max-prefill-tokens 32768  --chunked-prefill-size 32768 \
  --context-len 32768 \
  --max-running-requests 100 \
  --tp 8 --trust-remote-code

Node2:

MOONCAKE_CONFIG_PATH=./mooncake.json \
  python -m sglang.launch_server \
  --model-path my_model_path \
  --disaggregation-mode decode \
  --port 30000 --host 192.168.1.2 \
  --max-running-requests 100 \
  --tp 8 --trust-remote-code
python -m sglang.srt.disaggregation.mini_lb \
  --prefill http://192.168.1.1:30000 \
  --decode http://192.168.1.2:30000 \
  --host 0.0.0.0 --port 8000
python -m sglang.bench_serving --backend sglang   \
  --dataset-name random   --dataset-path /home/ShareGPT_V3_unfiltered_cleaned_split/ShareGPT_V3_unfiltered_cleaned_split.json  \
  --random-input 4096   --random-output 2048   --random-range-ratio 1   \
  --num-prompts 200  --max-concurrency 200   \
  --host 127.0.0.1  --port 8000   \
  --output-file "out.jsonl"  \
  --pd-seperated --flush-cache

Environment

  • 2 H800*8 nodes
  • RDMA*4 for each node
  • sglang main branch, commit id: 6aca583420490d8bebafc6239085b49f78325c0c
  • mooncake-transfer-engine 0.3.0b3

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 35 (2 by maintainers)

Most upvoted comments

@ShangmingCai The token usage on the prefill is 0.4, on decode is up to 0.8. You can check the full logs here.

prefill.log decode.log

@ShangmingCai I found the transfer length of engine.transfer_sync decreased to 256, I think that’s why the transfer slows down. Initially the length was about 1048576. Any idea why the data was not grouped together for the last 50-70 reqs?

sglang/python/sglang/srt/disaggregation/mooncake/conn.py

Line 166 in 85ec044

status = self.engine.transfer_sync(

If that is the case, then I think your token usage on the prefill node must be high? I think it could potentially be caused by the fragmentation issue of kvcache. The indices are not contiguous, so we cannot transfer them by combing them into one block, and transfer them indice by indice instead, this could be the reason why you got a slowdown. Let me run some tests to check it out.

Even I set --request-rate to 1.0 or 4.0, the result is the same. The last 50-70 reqs queued in the inflight queue, and transfer very slowly.

Oh, if they are in the inflight queue, then that could be bottlenecked by the transfer engine. Which model are you using? We will try to reproduce the results to see if it is a bug.