sglang: [Bug] 1.5B is bizarrely OOM on 80G A100
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
Simply specify the following parameters, and an error occurs during inference with qwen2.5-1.5b-instruct:
llm = sgl.Engine(model_path=m, dp_size=device_count())
{
"max_new_tokens": 512,
"temperature": 0.0,
"stop": [
"}\n",
"<|endoftext|>",
"<|im_end|>",
"</s>",
"## Question:",
"<|eot_id|>",
"\n\nQuestion:",
],
}
Traceback (most recent call last):
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 111, in forward_thread_func
with torch.get_device_module(self.device).stream(self.forward_stream):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/cuda/__init__.py", line 595, in __exit__
torch.cuda.set_stream(self.src_prev_stream) # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/vllm/utils.py", line 928, in _patched_set_stream
prev_set_stream(stream)
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/cuda/__init__.py", line 636, in set_stream
_set_stream_by_id(
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/cuda/__init__.py", line 618, in _set_stream_by_id
torch._C._cuda_setStream(
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
[2025-03-18 16:10:06 DP1 TP0] TpModelWorkerClient hit an exception: Traceback (most recent call last):
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 112, in forward_thread_func
self.forward_thread_func_()
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 143, in forward_thread_func_
logits_output, next_token_ids = self.worker.forward_batch_generation(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 172, in forward_batch_generation
logits_output = self.model_runner.forward(forward_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 946, in forward
return self.forward_decode(forward_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 897, in forward_decode
return self.model.forward(
^^^^^^^^^^^^^^^^^^^
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/models/qwen2.py", line 375, in forward
return self.logits_processor(
^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/layers/logits_processor.py", line 307, in forward
logits = self._get_logits(pruned_states, lm_head, logits_metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/env/rjh/lib/python3.12/site-packages/sglang/srt/layers/logits_processor.py", line 440, in _get_logits
logits = logits[:, : self.config.vocab_size].float()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.99 GiB. GPU 1 has a total capacity of 79.35 GiB of which 1.37 GiB is free. Process 28367 has 77.97 GiB memory in use. Of the allocated memory 74.20 GiB is allocated by PyTorch, with 30.52 MiB allocated in private pools (e.g., CUDA Graphs), and 1.26 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Reproduction
.
Environment
Python: 3.12.9 | packaged by conda-forge | (main, Feb 14 2025, 08:00:06) [GCC 13.3.0] CUDA available: True GPU 0,1,2,3: NVIDIA Graphics Device GPU 0,1,2,3 Compute Capability: 8.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.1, V12.1.105 CUDA Driver Version: 470.103.01 PyTorch: 2.5.1+cu121 sglang: 0.4.4.post1 sgl_kernel: 0.0.5.post3 flashinfer: 0.2.3+cu121torch2.5 triton: 3.1.0 transformers: 4.49.0 torchao: 0.7.0 numpy: 1.26.4 aiohttp: 3.11.9 fastapi: 0.115.6 hf_transfer: Module Not Found huggingface_hub: 0.26.3 interegular: 0.3.3 modelscope: Module Not Found orjson: 3.10.12 packaging: 24.0 psutil: 6.1.0 pydantic: 2.10.3 multipart: 0.0.20 zmq: 26.2.0 uvicorn: 0.32.1 uvloop: 0.21.0 vllm: 0.7.3 xgrammar: Module Not Found openai: 1.56.2 anthropic: Module Not Found litellm: Module Not Found decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 mlx5_0 mlx5_1 mlx5_2 mlx5_3 mlx5_4 mlx5_5 mlx5_6 mlx5_7 CPU Affinity NUMA Affinity GPU0 X NV8 NV8 NV8 PXB PXB NODE NODE SYS SYS SYS SYS 0-63,128-191 0 GPU1 NV8 X NV8 NV8 PXB PXB NODE NODE SYS SYS SYS SYS 0-63,128-191 0 GPU2 NV8 NV8 X NV8 NODE NODE PXB PXB SYS SYS SYS SYS 0-63,128-191 0 GPU3 NV8 NV8 NV8 X SYS SYS SYS SYS PXB PXB NODE NODE 64-127,192-255 1 mlx5_0 PXB PXB NODE SYS X PIX NODE NODE SYS SYS SYS SYS mlx5_1 PXB PXB NODE SYS PIX X NODE NODE SYS SYS SYS SYS mlx5_2 NODE NODE PXB SYS NODE NODE X PIX SYS SYS SYS SYS mlx5_3 NODE NODE PXB SYS NODE NODE PIX X SYS SYS SYS SYS mlx5_4 SYS SYS SYS PXB SYS SYS SYS SYS X PIX NODE NODE mlx5_5 SYS SYS SYS PXB SYS SYS SYS SYS PIX X NODE NODE mlx5_6 SYS SYS SYS NODE SYS SYS SYS SYS NODE NODE X PIX mlx5_7 SYS SYS SYS NODE SYS SYS SYS SYS NODE NODE PIX X
Legend:
X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks
ulimit soft: 1048576
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 47 (4 by maintainers)
batch input: https://drive.google.com/file/d/1KdXNMZr-Jm25cldokEN8vrE1kQMWhlx1/view?usp=sharing
In the meantime, got another OOM for this exact batch
Hi, I‘ve talked about this more specific as below.
Luckily, I found a case that triggers this OOM. This means we don’t have to align the cumbersome code.