sglang: [Bug] Qwen2.5-VL-7B-Instruct Inference Server crashes

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

This error does not happen every time, but it keeps crashing the Inference Server.

[2025-03-07 02:01:18 TP0] Scheduler hit an exception: Traceback (most recent call last): File “/home/username/learning/sglang/python/sglang/srt/managers/scheduler.py”, line 2290, in run_scheduler_process scheduler.event_loop_normal()
File “/home/username/learning/sglang/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py”, line 116, in decorate_context return func(*args, **kwargs)
File “/home/username/learning/sglang/python/sglang/srt/managers/scheduler.py”, line 486, in event_loop_normal result = self.run_batch(batch)
File “/home/username/learning/sglang/python/sglang/srt/managers/scheduler.py”, line 1187, in run_batch
logits_output, next_token_ids = self.tp_worker.forward_batch_generation(
File “/home/username/learning/sglang/python/sglang/srt/managers/tp_worker.py”, line 172, in forward_batch_generation logits_output = self.model_runner.forward(forward_batch)
File “/home/username/learning/sglang/python/sglang/srt/model_executor/model_runner.py”, line 909, in forward return self.forward_extend(
File “/home/username/learning/sglang/python/sglang/srt/model_executor/model_runner.py”, line 870, in forward_extend return self.model.forward(
File “/home/username/learning/sglang/python/sglang/srt/models/qwen2_5_vl.py”, line 671, in forward inputs_embeds[left_idx:right_idx, …, start_dim:end_dim] = (
RuntimeError: The expanded size of the tensor (24) must match the existing size (64) at non-singleton dimension 0. Target sizes: [24, 3584]. Tensor sizes: [64, 3584]

Reproduction

python3 -m sglang.launch_server --host 0.0.0.0 --port 30000 --model-path Qwen/Qwen2.5-VL-7B-Instruct --chat-template qwen2-vl

Environment

Running locally on Ubuntu 22.04. RTX 3090 Ti. Installation instructions are the latest.

pip install --upgrade pip
pip install uv
uv pip install "sglang[all]>=0.4.3.post4" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python

And also tried running the latest main branch, but kept getting the same error.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (1 by maintainers)

Most upvoted comments

I think disabling the prefill. --chunked-prefill-size -1 vanishes the error, but I’m not sure if this introduces some performance penalties.