sglang: [Bug] RuntimeError in ModelTpServer

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

benchmark serving got error when --num-prompts=5 num-prompts=1 and num-prompts=3 work fine, failed when num-prompts=5

Exception in ModelTpServer:
Traceback (most recent call last):
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 218, in exposed_step
    self.forward_step()
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 234, in forward_step
    self.forward_prefill_batch(new_batch)
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 446, in forward_prefill_batch
    output = self.model_runner.forward(batch, ForwardMode.EXTEND)
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 430, in forward
    return self.forward_extend(batch)
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 399, in forward_extend
    input_metadata = InputMetadata.from_schedule_batch(
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/model_executor/forward_batch_info.py", line 200, in from_schedule_batch
    ret.init_flashinfer_handlers(
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/model_executor/forward_batch_info.py", line 225, in init_flashinfer_handlers
    update_flashinfer_indices(
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/model_executor/forward_batch_info.py", line 315, in update_flashinfer_indices
    model_runner.flashinfer_prefill_wrapper_paged.begin_forward(
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/flashinfer/prefill.py", line 804, in begin_forward
    self._wrapper.begin_forward(
RuntimeError: Failed to allocate memory for batch_prefill_tmp_v with size 450494464 and alignment 16 in AlignedAllocator

Exception in ControllerSingle:
Traceback (most recent call last):
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/managers/controller_single.py", line 166, in start_controller_process
    controller.loop_for_forward()
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/managers/controller_single.py", line 103, in loop_for_forward
    out_pyobjs = self.tp_server.exposed_step(recv_reqs)
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 218, in exposed_step
    self.forward_step()
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 234, in forward_step
    self.forward_prefill_batch(new_batch)
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/managers/tp_worker.py", line 446, in forward_prefill_batch
    output = self.model_runner.forward(batch, ForwardMode.EXTEND)
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 430, in forward
    return self.forward_extend(batch)
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/model_executor/model_runner.py", line 399, in forward_extend
    input_metadata = InputMetadata.from_schedule_batch(
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/model_executor/forward_batch_info.py", line 200, in from_schedule_batch
    ret.init_flashinfer_handlers(
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/model_executor/forward_batch_info.py", line 225, in init_flashinfer_handlers
    update_flashinfer_indices(
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/sglang/srt/model_executor/forward_batch_info.py", line 315, in update_flashinfer_indices
    model_runner.flashinfer_prefill_wrapper_paged.begin_forward(
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/sglang-0.2.13/lib/python3.10/site-packages/flashinfer/prefill.py", line 804, in begin_forward
    self._wrapper.begin_forward(
RuntimeError: Failed to allocate memory for batch_prefill_tmp_v with size 450494464 and alignment 16 in AlignedAllocator

Reproduction

server:

python -m sglang.launch_server --model-path Qwen/Qwen2-7B-Instruct --tensor-parallel-size 1 --disable-radix-cache --host 0.0.0.0 --port 8083  --trust-remote-code --served-model-name Qwen2-7B-Instruct --mem-fraction-static 0.9

benchmark:

python benchmark_serving.py --model Qwen2-7B-Instruct --tokenizer Qwen/Qwen2-7B-Instruct  --host 0.0.0.0 --port 8083 --dataset-name random --random-input-len 4096 --random-output-len 256 --save-result --num-prompts 5

Environment

Python: 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA A100-SXM4-80GB GPU 0,1,2,3,4,5,6,7 Compute Capability: 8.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.1, V12.1.105 CUDA Driver Version: 525.85.12 PyTorch: 2.4.0+cu121 sglang: 0.2.13 flashinfer: 0.1.5+cu121torch2.4 triton: 3.0.0 transformers: 4.44.0 requests: 2.32.3 tqdm: 4.66.5 numpy: 1.26.4 aiohttp: 3.10.5 fastapi: 0.112.1 hf_transfer: 0.1.8 huggingface_hub: 0.24.6 interegular: 0.3.3 packaging: 24.1 PIL: 10.4.0 psutil: 6.0.0 pydantic: 2.8.2 uvicorn: 0.30.6 uvloop: 0.20.0 zmq: 26.1.1 vllm: 0.5.4 multipart: 0.0.9 openai: 1.41.1 anthropic: 0.34.1 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU0 X NV12 NV12 NV12 NV12 NV12 NV12 NV12 PXB PXBNODE NODE SYS SYS SYS SYS SYS 0-31,64-95 0 GPU1 NV12 X NV12 NV12 NV12 NV12 NV12 NV12 PXB PXBNODE NODE SYS SYS SYS SYS SYS 0-31,64-95 0 GPU2 NV12 NV12 X NV12 NV12 NV12 NV12 NV12 NODE NODEPXB PXB SYS SYS SYS SYS SYS 0-31,64-95 0 GPU3 NV12 NV12 NV12 X NV12 NV12 NV12 NV12 NODE NODEPXB PXB SYS SYS SYS SYS SYS 0-31,64-95 0 GPU4 NV12 NV12 NV12 NV12 X NV12 NV12 NV12 SYS SYSSYS SYS PXB PXB NODE NODE NODE 32-63,96-127 1 GPU5 NV12 NV12 NV12 NV12 NV12 X NV12 NV12 SYS SYSSYS SYS PXB PXB NODE NODE NODE 32-63,96-127 1 GPU6 NV12 NV12 NV12 NV12 NV12 NV12 X NV12 SYS SYSSYS SYS NODE NODE PXB PXB NODE 32-63,96-127 1 GPU7 NV12 NV12 NV12 NV12 NV12 NV12 NV12 X SYS SYSSYS SYS NODE NODE PXB PXB NODE 32-63,96-127 1 NIC0 PXB PXB NODE NODE SYS SYS SYS SYS X PIXNODE NODE SYS SYS SYS SYS SYS NIC1 PXB PXB NODE NODE SYS SYS SYS SYS PIX X NODE NODE SYS SYS SYS SYS SYS NIC2 NODE NODE PXB PXB SYS SYS SYS SYS NODE NODE X PIX SYS SYS SYS SYS SYS NIC3 NODE NODE PXB PXB SYS SYS SYS SYS NODE NODEPIX X SYS SYS SYS SYS SYS NIC4 SYS SYS SYS SYS PXB PXB NODE NODE SYS SYSSYS SYS X PIX NODE NODE NODE NIC5 SYS SYS SYS SYS PXB PXB NODE NODE SYS SYSSYS SYS PIX X NODE NODE NODE NIC6 SYS SYS SYS SYS NODE NODE PXB PXB SYS SYSSYS SYS NODE NODE X PIX NODE NIC7 SYS SYS SYS SYS NODE NODE PXB PXB SYS SYSSYS SYS NODE NODE PIX X NODE NIC8 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYSSYS SYS NODE NODE NODE NODE X

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_6 NIC5: mlx5_7 NIC6: mlx5_8 NIC7: mlx5_9 NIC8: mlx5_bond_0

ulimit soft: 65536

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 20 (9 by maintainers)

Commits related to this issue

Most upvoted comments

None of these results match the quoted results from their HF pages, however Hellaswag achieves very poor results.

@dmakhervaks The issue is fixed in https://github.com/sgl-project/sglang/pull/1428.