sglang: [Bug] 1.5B is bizarrely OOM on 80G A100

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

Simply specify the following parameters, and an error occurs during inference with qwen2.5-1.5b-instruct:

llm = sgl.Engine(model_path=m, dp_size=device_count())
{
    "max_new_tokens": 512,
    "temperature": 0.0,
    "stop": [
        "}\n",
        "<|endoftext|>",
        "<|im_end|>",
        "</s>",
        "## Question:",
        "<|eot_id|>",
        "\n\nQuestion:",
    ],
}
Traceback (most recent call last):
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 111, in forward_thread_func
    with torch.get_device_module(self.device).stream(self.forward_stream):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/cuda/__init__.py", line 595, in __exit__
    torch.cuda.set_stream(self.src_prev_stream)  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/vllm/utils.py", line 928, in _patched_set_stream
    prev_set_stream(stream)
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/cuda/__init__.py", line 636, in set_stream
    _set_stream_by_id(
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/cuda/__init__.py", line 618, in _set_stream_by_id
    torch._C._cuda_setStream(
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


[2025-03-18 16:10:06 DP1 TP0] TpModelWorkerClient hit an exception: Traceback (most recent call last):
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 112, in forward_thread_func
    self.forward_thread_func_()
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 143, in forward_thread_func_
    logits_output, next_token_ids = self.worker.forward_batch_generation(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 172, in forward_batch_generation
    logits_output = self.model_runner.forward(forward_batch)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 946, in forward
    return self.forward_decode(forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 897, in forward_decode
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/models/qwen2.py", line 375, in forward
    return self.logits_processor(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/layers/logits_processor.py", line 307, in forward
    logits = self._get_logits(pruned_states, lm_head, logits_metadata)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/layers/logits_processor.py", line 440, in _get_logits
    logits = logits[:, : self.config.vocab_size].float()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.99 GiB. GPU 1 has a total capacity of 79.35 GiB of which 1.37 GiB is free. Process 28367 has 77.97 GiB memory in use. Of the allocated memory 74.20 GiB is allocated by PyTorch, with 30.52 MiB allocated in private pools (e.g., CUDA Graphs), and 1.26 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Reproduction

.

Environment

Python: 3.12.9 | packaged by conda-forge | (main, Feb 14 2025, 08:00:06) [GCC 13.3.0] CUDA available: True GPU 0,1,2,3: NVIDIA Graphics Device GPU 0,1,2,3 Compute Capability: 8.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.1, V12.1.105 CUDA Driver Version: 470.103.01 PyTorch: 2.5.1+cu121 sglang: 0.4.4.post1 sgl_kernel: 0.0.5.post3 flashinfer: 0.2.3+cu121torch2.5 triton: 3.1.0 transformers: 4.49.0 torchao: 0.7.0 numpy: 1.26.4 aiohttp: 3.11.9 fastapi: 0.115.6 hf_transfer: Module Not Found huggingface_hub: 0.26.3 interegular: 0.3.3 modelscope: Module Not Found orjson: 3.10.12 packaging: 24.0 psutil: 6.1.0 pydantic: 2.10.3 multipart: 0.0.20 zmq: 26.2.0 uvicorn: 0.32.1 uvloop: 0.21.0 vllm: 0.7.3 xgrammar: Module Not Found openai: 1.56.2 anthropic: Module Not Found litellm: Module Not Found decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 mlx5_0 mlx5_1 mlx5_2 mlx5_3 mlx5_4 mlx5_5 mlx5_6 mlx5_7 CPU Affinity NUMA Affinity GPU0 X NV8 NV8 NV8 PXB PXB NODE NODE SYS SYS SYS SYS 0-63,128-191 0 GPU1 NV8 X NV8 NV8 PXB PXB NODE NODE SYS SYS SYS SYS 0-63,128-191 0 GPU2 NV8 NV8 X NV8 NODE NODE PXB PXB SYS SYS SYS SYS 0-63,128-191 0 GPU3 NV8 NV8 NV8 X SYS SYS SYS SYS PXB PXB NODE NODE 64-127,192-255 1 mlx5_0 PXB PXB NODE SYS X PIX NODE NODE SYS SYS SYS SYS mlx5_1 PXB PXB NODE SYS PIX X NODE NODE SYS SYS SYS SYS mlx5_2 NODE NODE PXB SYS NODE NODE X PIX SYS SYS SYS SYS mlx5_3 NODE NODE PXB SYS NODE NODE PIX X SYS SYS SYS SYS mlx5_4 SYS SYS SYS PXB SYS SYS SYS SYS X PIX NODE NODE mlx5_5 SYS SYS SYS PXB SYS SYS SYS SYS PIX X NODE NODE mlx5_6 SYS SYS SYS NODE SYS SYS SYS SYS NODE NODE X PIX mlx5_7 SYS SYS SYS NODE SYS SYS SYS SYS NODE NODE PIX X

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

ulimit soft: 1048576

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 47 (4 by maintainers)

Most upvoted comments

services:
  sglang:
    image: lmsysorg/sglang:latest
    container_name: sglang
    volumes:
      - ${HOME}/.cache/huggingface:/root/.cache/huggingface
      - /mnt/disks/sglang/data:/sglang/data
    restart: always
    ports:
       - 80:30000
    entrypoint: python3 -m sglang.launch_server
    command: --model-path allenai/olmOCR-7B-0225-preview
      --host 0.0.0.0
      --port 30000
      --chat-template qwen2-vl
      --file-storage-path /sglang/data
    ulimits:
      memlock: -1
      stack: 67108864
    ipc: host
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:30000/health || exit 1"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

batch input: https://drive.google.com/file/d/1KdXNMZr-Jm25cldokEN8vrE1kQMWhlx1/view?usp=sharing

In the meantime, got another OOM for this exact batch

sglang  | [2025-03-25 06:11:57] Child process unexpectedly failed with an exit code 9. pid=57

@rangehow I think you can try to adjust --mem-fraction-static (the default value is 0.9).

Hi, I‘ve talked about this more specific as below.

I can provide some additional information: Even if I reduce the kv cache to 0.6, I still encounter an OOM (Out of Memory) issue. However, setting the batch size (bsz) to a fixed value—16 in my case—works (with 40% utilization).

Luckily, I found a case that triggers this OOM. This means we don’t have to align the cumbersome code.

lm_eval --model sglang \
    --model_args pretrained=Qwen2.5-1.5B-Instruct/main,dp_size=4,dtype=auto,mem_fraction_static=0.8\
    --tasks mmlu \
    --batch_size auto