sglang: [Bug] When deploying QWEN3 using PD separation and the DP + EP + DeepEP method, the Decode instance get stuck.
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
When deploying QWEN3 using PD separation and the DP + EP + DeepEP method, the Decode instance of sglang will get stuck.
When the program is evaluated with an input of 1024 and an output of 1024, it runs normally. However, when the input is 2048 and the output is 2048, it gets stuck.
@zhyncs @ShangmingCai @fzyzcjy
Reproduction
2P
python3 -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-FP8 --load-format dummy --trust-remote-code --disaggregation-mode prefill --dist-init-addr 10.130.8.138:40050 --nnodes 2 --node-rank 0 --tp 16 --dp 16 --enable-dp-attention --enable-deepep-moe --ep-size 16 --disable-radix-cache --mem-frac 0.8 --deepep-mode normal --host 0.0.0.0 --moe-dense-tp-size 1 --enable-dp-lm-head --watchdog-timeout 1000000 --decode-log-interval 1 --context-length 1000000
python3 -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-FP8 --load-format dummy --trust-remote-code --disaggregation-mode prefill --dist-init-addr 10.130.8.138:40050 --nnodes 2 --node-rank 1 --tp 16 --dp 16 --enable-dp-attention --enable-deepep-moe --ep-size 16 --disable-radix-cache --mem-frac 0.8 --deepep-mode normal --host 0.0.0.0 --moe-dense-tp-size 1 --enable-dp-lm-head --watchdog-timeout 1000000 --decode-log-interval 1 --context-length 1000000
4D
SGLANG_TORCH_PROFILER_DIR=~/logs python3 -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-FP8 --load-format dummy --trust-remote-code --disaggregation-mode decode --dist-init-addr 10.130.8.143:8050 --nnodes 4 --node-rank 0 --tp 32 --dp 32 --enable-dp-attention --enable-deepep-moe --ep-size 32 --disable-radix-cache --mem-frac 0.8 --deepep-mode low_latency --host 0.0.0.0 --moe-dense-tp-size 1 --enable-dp-lm-head --cuda-graph-max-bs 128 --watchdog-timeout 1000000 --decode-log-interval 1 --context-length 1000000
SGLANG_TORCH_PROFILER_DIR=~/logs python3 -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-FP8 --load-format dummy --trust-remote-code --disaggregation-mode decode --dist-init-addr 10.130.8.143:8050 --nnodes 4 --node-rank 1 --tp 32 --dp 32 --enable-dp-attention --enable-deepep-moe --ep-size 32 --disable-radix-cache --mem-frac 0.8 --deepep-mode low_latency --host 0.0.0.0 --moe-dense-tp-size 1 --enable-dp-lm-head --cuda-graph-max-bs 128 --watchdog-timeout 1000000 --decode-log-interval 1 --context-length 1000000
SGLANG_TORCH_PROFILER_DIR=~/logs python3 -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-FP8 --load-format dummy --trust-remote-code --disaggregation-mode decode --dist-init-addr 10.130.8.143:8050 --nnodes 4 --node-rank 2 --tp 32 --dp 32 --enable-dp-attention --enable-deepep-moe --ep-size 32 --disable-radix-cache --mem-frac 0.8 --deepep-mode low_latency --host 0.0.0.0 --moe-dense-tp-size 1 --enable-dp-lm-head --cuda-graph-max-bs 128 --watchdog-timeout 1000000 --decode-log-interval 1 --context-length 1000000
SGLANG_TORCH_PROFILER_DIR=~/logs python3 -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-FP8 --load-format dummy --trust-remote-code --disaggregation-mode decode --dist-init-addr 10.130.8.143:8050 --nnodes 4 --node-rank 3 --tp 32 --dp 32 --enable-dp-attention --enable-deepep-moe --ep-size 32 --disable-radix-cache --mem-frac 0.8 --deepep-mode low_latency --host 0.0.0.0 --moe-dense-tp-size 1 --enable-dp-lm-head --cuda-graph-max-bs 128 --watchdog-timeout 1000000 --decode-log-interval 1 --context-length 1000000
Proxy
python3 -m sglang.srt.disaggregation.mini_lb --prefill "http://10.130.8.138:30000" --decode "http://10.130.8.143:30000" --host 0.0.0.0
Benchmark
python3 -m sglang.bench_serving --backend sglang --dataset-name random --num-prompt 3200 --request-rate 16 --random-input 2048 --random-output 2048 --model Qwen/Qwen3-235B-A22B-FP8 --port 8000
Environment
Python: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb 6 2025, 18:56:27) [GCC 11.2.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA L20Y GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.4, V12.4.99 CUDA Driver Version: 550.127.08 PyTorch: 2.6.0+cu124 sglang: 0.4.6.post5 sgl_kernel: 0.1.5 flashinfer_python: 0.2.5 triton: 3.2.0 transformers: 4.52.3 torchao: 0.9.0 numpy: 2.2.6 aiohttp: 3.12.6 fastapi: 0.115.12 hf_transfer: 0.1.9 huggingface_hub: 0.32.3 interegular: 0.3.3 modelscope: 1.26.0 orjson: 3.10.18 outlines: 0.1.11 packaging: 25.0 psutil: 7.0.0 pydantic: 2.11.5 python-multipart: 0.0.20 pyzmq: 26.4.0 uvicorn: 0.34.3 uvloop: 0.21.0 vllm: Module Not Found xgrammar: 0.1.19 openai: 1.82.1 tiktoken: 0.9.0 anthropic: 0.52.1 litellm: 1.72.0 decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV8 NV8 NV8 NV8 NV8 NV8 NV8 PIX NODE NODE NODE SYS SYS SYS SYS 0,2,4,6,8,10 0 N/A GPU1 NV8 X NV8 NV8 NV8 NV8 NV8 NV8 NODE PIX NODE NODE SYS SYS SYS SYS 0,2,4,6,8,10 0 N/A GPU2 NV8 NV8 X NV8 NV8 NV8 NV8 NV8 NODE NODE PIX NODE SYS SYS SYS SYS 0,2,4,6,8,10 0 N/A GPU3 NV8 NV8 NV8 X NV8 NV8 NV8 NV8 NODE NODE NODE PIX SYS SYS SYS SYS 0,2,4,6,8,10 0 N/A GPU4 NV8 NV8 NV8 NV8 X NV8 NV8 NV8 SYS SYS SYS SYS PIX NODE NODE NODE 1,3,5,7,9,11 1 N/A GPU5 NV8 NV8 NV8 NV8 NV8 X NV8 NV8 SYS SYS SYS SYS NODE PIX NODE NODE 1,3,5,7,9,11 1 N/A GPU6 NV8 NV8 NV8 NV8 NV8 NV8 X NV8 SYS SYS SYS SYS NODE NODE PIX NODE 1,3,5,7,9,11 1 N/A GPU7 NV8 NV8 NV8 NV8 NV8 NV8 NV8 X SYS SYS SYS SYS NODE NODE NODE PIX 1,3,5,7,9,11 1 N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS NIC1 NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS NIC2 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE SYS SYS SYS SYS NIC3 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X SYS SYS SYS SYS NIC4 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE NIC5 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE NIC6 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE NIC7 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X
Legend:
X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_bond_0 NIC1: mlx5_bond_1 NIC2: mlx5_bond_2 NIC3: mlx5_bond_3 NIC4: mlx5_bond_4 NIC5: mlx5_bond_5 NIC6: mlx5_bond_6 NIC7: mlx5_bond_7
ulimit soft: 65535
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 19
did you use py-spy to dump the stack for profile/decode, was the same as previous? And I tried it for 10,000 requests but can not reproduce it, I will try more times.