sglang: [Bug] Qwen2.5-VL-7B-Instruct Inference Server crashes
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
This error does not happen every time, but it keeps crashing the Inference Server.
[2025-03-07 02:01:18 TP0] Scheduler hit an exception: Traceback (most recent call last):
File “/home/username/learning/sglang/python/sglang/srt/managers/scheduler.py”, line 2290, in run_scheduler_process
scheduler.event_loop_normal()
File “/home/username/learning/sglang/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py”, line 116, in decorate_context
return func(*args, **kwargs)
File “/home/username/learning/sglang/python/sglang/srt/managers/scheduler.py”, line 486, in event_loop_normal
result = self.run_batch(batch)
File “/home/username/learning/sglang/python/sglang/srt/managers/scheduler.py”, line 1187, in run_batch
logits_output, next_token_ids = self.tp_worker.forward_batch_generation(
File “/home/username/learning/sglang/python/sglang/srt/managers/tp_worker.py”, line 172, in forward_batch_generation
logits_output = self.model_runner.forward(forward_batch)
File “/home/username/learning/sglang/python/sglang/srt/model_executor/model_runner.py”, line 909, in forward
return self.forward_extend(
File “/home/username/learning/sglang/python/sglang/srt/model_executor/model_runner.py”, line 870, in forward_extend
return self.model.forward(
File “/home/username/learning/sglang/python/sglang/srt/models/qwen2_5_vl.py”, line 671, in forward
inputs_embeds[left_idx:right_idx, …, start_dim:end_dim] = (
RuntimeError: The expanded size of the tensor (24) must match the existing size (64) at non-singleton dimension 0. Target sizes: [24, 3584]. Tensor sizes: [64, 3584]
Reproduction
python3 -m sglang.launch_server --host 0.0.0.0 --port 30000 --model-path Qwen/Qwen2.5-VL-7B-Instruct --chat-template qwen2-vl
Environment
Running locally on Ubuntu 22.04. RTX 3090 Ti. Installation instructions are the latest.
pip install --upgrade pip
pip install uv
uv pip install "sglang[all]>=0.4.3.post4" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python
And also tried running the latest main branch, but kept getting the same error.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (1 by maintainers)
I think disabling the prefill.
--chunked-prefill-size -1vanishes the error, but I’m not sure if this introduces some performance penalties.