sglang: [Bug] Performance Regression: Eagle Speculative Decoding Causes Significant Inference Slowdown

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

target model : qwq-32b

without eagle:

[2025-06-07 21:25:14 TP0] Decode batch. #running-req: 1, #token: 9322, token usage: 0.00, gen throughput (token/s): 97.52, #queue-req: 0
[2025-06-07 21:25:14 TP0] Decode batch. #running-req: 1, #token: 9362, token usage: 0.00, gen throughput (token/s): 97.58, #queue-req: 0
[2025-06-07 21:25:15 TP0] Decode batch. #running-req: 1, #token: 9402, token usage: 0.00, gen throughput (token/s): 97.63, #queue-req: 0
[2025-06-07 21:25:15 TP0] Decode batch. #running-req: 1, #token: 9442, token usage: 0.00, gen throughput (token/s): 97.50, #queue-req: 0
[2025-06-07 21:25:15 TP0] Decode batch. #running-req: 1, #token: 9482, token usage: 0.00, gen throughput (token/s): 97.55, #queue-req: 0
[2025-06-07 21:25:16 TP0] Decode batch. #running-req: 1, #token: 9522, token usage: 0.00, gen throughput (token/s): 97.54, #queue-req: 0
[2025-06-07 21:25:16 TP0] Decode batch. #running-req: 1, #token: 9562, token usage: 0.00, gen throughput (token/s): 97.61, #queue-req: 0
[2025-06-07 21:25:17 TP0] Decode batch. #running-req: 1, #token: 9602, token usage: 0.00, gen throughput (token/s): 97.57, #queue-req: 0
[2025-06-07 21:25:17 TP0] Decode batch. #running-req: 1, #token: 9642, token usage: 0.00, gen throughput (token/s): 97.54, #queue-req: 0
[2025-06-07 21:25:17 TP0] Decode batch. #running-req: 1, #token: 9682, token usage: 0.00, gen throughput (token/s): 97.53, #queue-req: 0
[2025-06-07 21:25:18 TP0] Decode batch. #running-req: 1, #token: 9722, token usage: 0.00, gen throughput (token/s): 97.55, #queue-req: 0
[2025-06-07 21:25:18 TP0] Decode batch. #running-req: 1, #token: 9762, token usage: 0.00, gen throughput (token/s): 97.50, #queue-req: 0

with eagle

[2025-06-07 21:21:09 TP0] Decode batch. #running-req: 1, #token: 9448, token usage: 0.00, accept len: 2.38, gen throughput (token/s): 79.63, #queue-req: 0
[2025-06-07 21:21:11 TP0] Decode batch. #running-req: 1, #token: 9537, token usage: 0.00, accept len: 2.23, gen throughput (token/s): 77.67, #queue-req: 0
[2025-06-07 21:21:12 TP0] Decode batch. #running-req: 1, #token: 9635, token usage: 0.00, accept len: 2.45, gen throughput (token/s): 84.50, #queue-req: 0
[2025-06-07 21:21:13 TP0] Decode batch. #running-req: 1, #token: 9718, token usage: 0.00, accept len: 2.08, gen throughput (token/s): 72.10, #queue-req: 0
[2025-06-07 21:21:14 TP0] Decode batch. #running-req: 1, #token: 9815, token usage: 0.00, accept len: 2.42, gen throughput (token/s): 82.95, #queue-req: 0
[2025-06-07 21:21:15 TP0] Decode batch. #running-req: 1, #token: 9890, token usage: 0.00, accept len: 1.88, gen throughput (token/s): 65.86, #queue-req: 0
[2025-06-07 21:21:16 TP0] Decode batch. #running-req: 1, #token: 9979, token usage: 0.00, accept len: 2.23, gen throughput (token/s): 76.11, #queue-req: 0

### Reproduction

python -m sglang.launch_server --model-path {model_path} \
--trust-remote-code \
--tp 8 \
--enable-cache-report \
--log-level info \
--mem-fraction-static 0.8 \
--host 0.0.0.0 \
--port 9122 \
--speculative-algo EAGLE \
--speculative-draft {eagle_path} \
--speculative-num-steps 3 \
--speculative-eagle-topk 4 \
--speculative-num-draft-tokens 5

### Environment

Python: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H20
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
CUDA Driver Version: 535.183.06
PyTorch: 2.6.0+cu124
sglang: 0.4.6.post1
sgl_kernel: 0.1.0
flashinfer: Module Not Found
triton: 3.2.0
transformers: 4.51.3
torchao: 0.10.0
numpy: 1.26.4
aiohttp: 3.11.13
fastapi: 0.115.11
hf_transfer: Module Not Found
huggingface_hub: 0.30.2
interegular: 0.3.3
modelscope: Module Not Found
orjson: 3.10.15
outlines: 0.1.11
packaging: 24.2
psutil: 7.0.0
pydantic: 2.11.3
multipart: Module Not Found
zmq: Module Not Found
uvicorn: 0.34.0
uvloop: 0.21.0
vllm: 0.7.3.post1.dev5+g4754911ac.cu124.ant
xgrammar: 0.1.11
openai: 1.58.1
tiktoken: 0.9.0
anthropic: Module Not Found
litellm: Module Not Found
decord: 0.6.0
NVIDIA Topology: 
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    NODE    NODE    SYS     SYS     0-47,96-143     0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    SYS     SYS     0-47,96-143     0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    NODE    SYS     SYS     0-47,96-143     0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    PIX     SYS     SYS     0-47,96-143     0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     PIX     NODE    48-95,144-191   1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     NODE    NODE    48-95,144-191   1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     NODE    PIX     48-95,144-191   1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     NODE    NODE    48-95,144-191   1               N/A
NIC0    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    SYS     SYS
NIC1    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     NODE     X      SYS     SYS
NIC2    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    SYS     SYS      X      NODE
NIC3    SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    SYS     SYS     NODE     X 

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 18 (17 by maintainers)

Most upvoted comments

在我的 H100 GPU 上的 Eagle 模型上,tp=1,Qwen32 在不使用 Eagle 的情况下达到了近 60tps,使用 Eagle 的情况下则达到了 100-140tps 左右。尝试在社区发布的模型上测试性能,然后与您的模型进行比较。这样您就可以了解问题所在——是模型还是您的 GPU。

非常感谢您分享您的 H100 基准测试。启用 Eagle 后,您是否观察到 tp=8 时 tps 进一步提升?

由于我没有那么多 GPU,所以我没有在 tp=8 上测试。我们在 tp=2 上测试过,结果 Eagle 的效率下降非常剧烈。在 tp=1 上,结果还不错。

I think verify cost too much time. I profile Qwen3-30b-A3b with EAGLE3 on 1*H20. cmd

python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B --served-model-name Qwen3-30B --port 9122 --tensor-parallel-size 1 --mem-fraction-static 0.8 --max-running-requests 64 --trust-remote-code --enable-cache-report --cuda-graph-max-bs 16 --attention-backend flashinfer --speculative-algorithm EAGLE3 --speculative-draft-model-path qwen3_30b_moe_eagle3 --speculative-num-steps 6 --speculative-eagle-topk 10 --speculative-num-draft-tokens 15

With EAGLE3, draft cost 2.73ms, verify cost 17.16 ms, acc len = 2.5 Image

without spec, 7.96 ms

Image

Try running with these parameters (taken from my prod environment):

–attention-backend flashinfer
–quantization w8a8_fp8
–speculative-num-steps 3
–speculative-eagle-topk 2
–speculative-num-draft-tokens 4 \

Before that, also update sglang, install it directly from source code, because many eagle updates haven’t made it into the new version yet.

Try retraining the draft model using the Hass method https://github.com/HArmonizedSS/HASS - this gave us better results. This method simply improves model training and doesn’t affect inference.

Вы тренировали модель Eagle специально для QwQ? В открытом исходном коде нет такого чернового варианта модели.

Да, я думаю, что длина приема в порядке.

#6783 (comment)

The max_position_embeddings in base model and eagle model : 40k base model

{
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "hidden_act": "silu",
  "hidden_size": 5120,
  "initializer_range": 0.02,
  "intermediate_size": 27648,
  "max_position_embeddings": 40960,
  "max_window_layers": 64,
  "model_type": "qwen2",
  "num_attention_heads": 40,
  "num_hidden_layers": 64,
  "num_key_value_heads": 8,
  "pad_token_id": 151643,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": 32768,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.51.0",
  "use_cache": false,
  "use_sliding_window": false,
  "vocab_size": 152064
}

eagle model

{
    "architectures": [
        "Qwen2ForCausalLMEagle"
    ],
    "attention_dropout": 0.0,
    "bos_token_id": 151643,
    "eos_token_id": 151645,
    "hidden_act": "silu",
    "hidden_size": 5120,
    "initializer_range": 0.02,
    "intermediate_size": 27648,
    "max_position_embeddings": 40960,
    "max_window_layers": 64,
    "model_type": "qwen2",
    "num_attention_heads": 40,
    "num_hidden_layers": 1,
    "num_key_value_heads": 8,
    "pad_token_id": 151643,
    "rms_norm_eps": 1e-05,
    "rope_theta": 1000000.0,
    "sliding_window": 32768,
    "tie_word_embeddings": false,
    "torch_dtype": "bfloat16",
    "transformers_version": "4.44.2",
    "use_cache": true,
    "use_sliding_window": false,
    "vocab_size": 152064,
    "qkv_bias": true,
    "bias": false
}