sglang: [Bug] Performance Regression: Eagle Speculative Decoding Causes Significant Inference Slowdown
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
target model : qwq-32b
without eagle:
[2025-06-07 21:25:14 TP0] Decode batch. #running-req: 1, #token: 9322, token usage: 0.00, gen throughput (token/s): 97.52, #queue-req: 0
[2025-06-07 21:25:14 TP0] Decode batch. #running-req: 1, #token: 9362, token usage: 0.00, gen throughput (token/s): 97.58, #queue-req: 0
[2025-06-07 21:25:15 TP0] Decode batch. #running-req: 1, #token: 9402, token usage: 0.00, gen throughput (token/s): 97.63, #queue-req: 0
[2025-06-07 21:25:15 TP0] Decode batch. #running-req: 1, #token: 9442, token usage: 0.00, gen throughput (token/s): 97.50, #queue-req: 0
[2025-06-07 21:25:15 TP0] Decode batch. #running-req: 1, #token: 9482, token usage: 0.00, gen throughput (token/s): 97.55, #queue-req: 0
[2025-06-07 21:25:16 TP0] Decode batch. #running-req: 1, #token: 9522, token usage: 0.00, gen throughput (token/s): 97.54, #queue-req: 0
[2025-06-07 21:25:16 TP0] Decode batch. #running-req: 1, #token: 9562, token usage: 0.00, gen throughput (token/s): 97.61, #queue-req: 0
[2025-06-07 21:25:17 TP0] Decode batch. #running-req: 1, #token: 9602, token usage: 0.00, gen throughput (token/s): 97.57, #queue-req: 0
[2025-06-07 21:25:17 TP0] Decode batch. #running-req: 1, #token: 9642, token usage: 0.00, gen throughput (token/s): 97.54, #queue-req: 0
[2025-06-07 21:25:17 TP0] Decode batch. #running-req: 1, #token: 9682, token usage: 0.00, gen throughput (token/s): 97.53, #queue-req: 0
[2025-06-07 21:25:18 TP0] Decode batch. #running-req: 1, #token: 9722, token usage: 0.00, gen throughput (token/s): 97.55, #queue-req: 0
[2025-06-07 21:25:18 TP0] Decode batch. #running-req: 1, #token: 9762, token usage: 0.00, gen throughput (token/s): 97.50, #queue-req: 0
with eagle
[2025-06-07 21:21:09 TP0] Decode batch. #running-req: 1, #token: 9448, token usage: 0.00, accept len: 2.38, gen throughput (token/s): 79.63, #queue-req: 0
[2025-06-07 21:21:11 TP0] Decode batch. #running-req: 1, #token: 9537, token usage: 0.00, accept len: 2.23, gen throughput (token/s): 77.67, #queue-req: 0
[2025-06-07 21:21:12 TP0] Decode batch. #running-req: 1, #token: 9635, token usage: 0.00, accept len: 2.45, gen throughput (token/s): 84.50, #queue-req: 0
[2025-06-07 21:21:13 TP0] Decode batch. #running-req: 1, #token: 9718, token usage: 0.00, accept len: 2.08, gen throughput (token/s): 72.10, #queue-req: 0
[2025-06-07 21:21:14 TP0] Decode batch. #running-req: 1, #token: 9815, token usage: 0.00, accept len: 2.42, gen throughput (token/s): 82.95, #queue-req: 0
[2025-06-07 21:21:15 TP0] Decode batch. #running-req: 1, #token: 9890, token usage: 0.00, accept len: 1.88, gen throughput (token/s): 65.86, #queue-req: 0
[2025-06-07 21:21:16 TP0] Decode batch. #running-req: 1, #token: 9979, token usage: 0.00, accept len: 2.23, gen throughput (token/s): 76.11, #queue-req: 0
### Reproduction
python -m sglang.launch_server --model-path {model_path} \
--trust-remote-code \
--tp 8 \
--enable-cache-report \
--log-level info \
--mem-fraction-static 0.8 \
--host 0.0.0.0 \
--port 9122 \
--speculative-algo EAGLE \
--speculative-draft {eagle_path} \
--speculative-num-steps 3 \
--speculative-eagle-topk 4 \
--speculative-num-draft-tokens 5
### Environment
Python: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H20
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
CUDA Driver Version: 535.183.06
PyTorch: 2.6.0+cu124
sglang: 0.4.6.post1
sgl_kernel: 0.1.0
flashinfer: Module Not Found
triton: 3.2.0
transformers: 4.51.3
torchao: 0.10.0
numpy: 1.26.4
aiohttp: 3.11.13
fastapi: 0.115.11
hf_transfer: Module Not Found
huggingface_hub: 0.30.2
interegular: 0.3.3
modelscope: Module Not Found
orjson: 3.10.15
outlines: 0.1.11
packaging: 24.2
psutil: 7.0.0
pydantic: 2.11.3
multipart: Module Not Found
zmq: Module Not Found
uvicorn: 0.34.0
uvloop: 0.21.0
vllm: 0.7.3.post1.dev5+g4754911ac.cu124.ant
xgrammar: 0.1.11
openai: 1.58.1
tiktoken: 0.9.0
anthropic: Module Not Found
litellm: Module Not Found
decord: 0.6.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE SYS SYS 0-47,96-143 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE SYS SYS 0-47,96-143 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE SYS SYS 0-47,96-143 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE PIX SYS SYS 0-47,96-143 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS PIX NODE 48-95,144-191 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS NODE NODE 48-95,144-191 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS NODE PIX 48-95,144-191 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS NODE NODE 48-95,144-191 1 N/A
NIC0 NODE PIX NODE NODE SYS SYS SYS SYS X NODE SYS SYS
NIC1 NODE NODE NODE PIX SYS SYS SYS SYS NODE X SYS SYS
NIC2 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS X NODE
NIC3 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS NODE X
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 18 (17 by maintainers)
I think verify cost too much time. I profile Qwen3-30b-A3b with EAGLE3 on 1*H20. cmd
With EAGLE3, draft cost 2.73ms, verify cost 17.16 ms, acc len = 2.5
without spec, 7.96 ms
Try running with these parameters (taken from my prod environment):
–attention-backend flashinfer
–quantization w8a8_fp8
–speculative-num-steps 3
–speculative-eagle-topk 2
–speculative-num-draft-tokens 4 \
Before that, also update sglang, install it directly from source code, because many eagle updates haven’t made it into the new version yet.
Try retraining the draft model using the Hass method https://github.com/HArmonizedSS/HASS - this gave us better results. This method simply improves model training and doesn’t affect inference.
The
max_position_embeddingsin base model and eagle model : 40k base modeleagle model