sglang: [Bug] Model Stuck at Prefill and then throw "Watchdog Timeout" Error After Idle Period (Deepseek-r1:671b on two H100*8)
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
I am currently using SGLang to deploy the deepseek-r1:671b model across two H800 GPUs. However, I have encountered a persistent issue when the system remains idle for some time. Upon resuming usage, even with simple prompts such as “Hello,” the model gets stuck during the Prefill stage. Subsequently, the system throws a “watchdog timeout” error.
Following this error, the GPU resources are released, and any subsequent attempts to interact with the model fail to reload it. The only way to restore functionality is by restarting the service entirely.
Reproduction
Deploy the deepseek-r1:671b model using SGLang on two H800 GPUs.
docker run --gpus all \
--shm-size 32g \
--network=host \
-v /model/:/model \
--name sglang_multinode \
-it \
--rm \
--env "GLOO_SOCKET_IFNAME=ens24f0" \
--env "NCCL_SOCKET_IFNAME=ens24f0" \
--env "NCCL_DEBUG=INFO" \
--env "NCCL_IGNORE_DISABLED_P2P=1" \
--ipc=host \
lmsysorg/sglang:latest \
python3 -m sglang.launch_server --model-path /model/deepseek-ai/DeepSeek-R1 --tp 16 --dist-init-addr 10.2.17.101:20000 --nnodes 2 --node-rank 0 --trust-remote-code --host 0.0.0.0 --port 40000
It works at the begining. Leave the system idle for a period of time (exact duration may vary). Attempt to send a simple query like “Hello” after the idle period. Observe that the model gets stuck at the Prefill stage. Encounter the “watchdog timeout” error, followed by the release of GPU resources.
[2025-02-25 05:31:35] INFO: 10.2.17.101:38344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-02-25 05:31:35 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 5, cache hit rate: 42.12%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2025-02-25 05:32:28] INFO: 10.2.17.101:60714 - "GET /v1/models HTTP/1.1" 200 OK
[2025-02-25 05:36:52 TP4] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-25 05:36:53 TP7] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-25 05:36:53 TP2] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-25 05:36:54 TP1] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-25 05:36:54 TP3] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-25 05:36:54 TP5] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-25 05:36:54 TP0] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-25 05:36:55 TP6] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-25 05:36:57] Received sigquit from a child proces. It usually means the child failed.
[2025-02-25 05:36:57] Received sigquit from a child proces. It usually means the child failed.
[2025-02-25 05:36:57] Received sigquit from a child proces. It usually means the child failed.
[2025-02-25 05:36:57] Received sigquit from a child proces. It usually means the child failed.
.......
Note that further queries do not reload the model, necessitating a service restart.
Environment
Python: 3.10.12 (main, Jan 17 2025, 14:35:34) [GCC 11.4.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA H100 80GB HBM3 GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.4, V12.4.131 CUDA Driver Version: 550.144.03 PyTorch: 2.5.1+cu124 sgl_kernel: 0.0.3.post6 flashinfer: 0.2.1.post2+cu124torch2.5 triton: 3.1.0 transformers: 4.48.3 torchao: 0.8.0 numpy: 1.26.4 aiohttp: 3.11.12 fastapi: 0.115.8 hf_transfer: 0.1.9 huggingface_hub: 0.28.1 interegular: 0.3.3 modelscope: 1.23.0 orjson: 3.10.15 packaging: 24.2 psutil: 7.0.0 pydantic: 2.10.6 multipart: 0.0.20 zmq: 26.2.1 uvicorn: 0.34.0 uvloop: 0.21.0 vllm: 0.7.2 openai: 1.63.2 tiktoken: 0.9.0 anthropic: 0.45.2 decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE PIX SYS SYS SYS SYS 0-47,96-143 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS PIX NODE NODE NODE 48-95,144-191 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS NODE PIX NODE NODE 48-95,144-191 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS NODE NODE PIX NODE 48-95,144-191 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS NODE NODE NODE PIX 48-95,144-191 1 N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE SYS SYS SYS SYS NIC1 NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE SYS SYS SYS SYS NIC2 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE X SYS SYS SYS SYS NIC3 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS X NODE NODE NODE NIC4 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS NODE X NODE NODE NIC5 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS NODE NODE X NODE NIC6 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS NODE NODE NODE X
Legend:
X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6
ulimit soft: 1048576
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 57 (1 by maintainers)
@FrankLeeeee The problem still exists - the logs remain stuck at prefill when the service is idle for a while, but the service now exits after watchdog timeout. Can we get this issue rechecked or reopened? What logs/system data should I gather next ?
Hi guys, a silly question. What’s the difference between --dist-timeout 3600 and --watchdog_timeout 3600, and if we add --dist-timeout 3600 will the server hangs for 3599s at worst?
It starts to be normal, but after a few days , there will be such an error, is there a way to automatically restart?