sglang: [Bug] DeepSeek R1 serve crash occasionally on 2*H100

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

When deploying dpsk-r1 using IB communication on two H100 units, unexpected crashes frequently occur, with a probability of occurring under tests with different numbers of requests.

Reproduction

env installation

following the installation in https://docs.sglang.ai/start/install.html

pip install --upgrade pip
conda install -c nvidia/label/cuda-12.4.0 cuda=12.4.0
pip install sgl-kernel --force-reinstall --no-deps
pip install "sglang[all]>=0.4.2.post2" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer/

package information

aiohappyeyeballs==2.4.4
aiohttp==3.11.12
aiosignal==1.3.2
annotated-types==0.7.0
anthropic==0.45.2
anyio==4.8.0
asttokens==3.0.0
async-timeout==5.0.1
attrs==25.1.0
certifi==2025.1.31
charset-normalizer==3.4.1
click==8.1.8
cloudpickle==3.1.1
compressed-tensors==0.8.0
cuda-bindings==12.8.0
cuda-python==12.8.0
datasets==3.2.0
decorator==5.1.1
decord==0.6.0
dill==0.3.8
diskcache==5.6.3
distro==1.9.0
einops==0.8.0
exceptiongroup==1.2.2
executing==2.2.0
fastapi==0.115.8
filelock==3.17.0
flashinfer-python==0.2.0.post2+cu124torch2.5
frozenlist==1.5.0
fsspec==2024.9.0
gguf==0.10.0
h11==0.14.0
hf_transfer==0.1.9
httpcore==1.0.7
httptools==0.6.4
httpx==0.28.1
huggingface-hub==0.28.1
idna==3.10
importlib_metadata==8.6.1
iniconfig==2.0.0
interegular==0.3.3
ipython==8.32.0
jedi==0.19.2
Jinja2==3.1.5
jiter==0.8.2
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
lark==1.2.2
litellm==1.60.6
llvmlite==0.44.0
lm-format-enforcer==0.10.9
MarkupSafe==3.0.2
matplotlib-inline==0.1.7
mistral_common==1.5.2
modelscope==1.22.3
mpmath==1.3.0
msgpack==1.1.0
msgspec==0.19.0
multidict==6.1.0
multiprocess==0.70.16
nest-asyncio==1.6.0
networkx==3.4.2
numba==0.61.0
numpy==1.26.4
nvidia-cublas-cu12==12.4.5.8
nvidia-cuda-cupti-cu12==12.4.127
nvidia-cuda-nvrtc-cu12==12.4.127
nvidia-cuda-runtime-cu12==12.4.127
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.2.1.3
nvidia-curand-cu12==10.3.5.147
nvidia-cusolver-cu12==11.6.1.9
nvidia-cusparse-cu12==12.3.1.170
nvidia-ml-py==12.570.86
nvidia-nccl-cu12==2.21.5
nvidia-nvjitlink-cu12==12.4.127
nvidia-nvtx-cu12==12.4.127
openai==1.61.1
opencv-python-headless==4.11.0.86
orjson==3.10.15
outlines==0.0.46
packaging==24.2
pandas==2.2.3
parso==0.8.4
partial-json-parser==0.2.1.1.post5
pexpect==4.9.0
pillow==10.4.0
pluggy==1.5.0
prometheus-fastapi-instrumentator==7.0.2
prometheus_client==0.21.1
prompt_toolkit==3.0.50
propcache==0.2.1
protobuf==5.29.3
psutil==6.1.1
ptyprocess==0.7.0
pure_eval==0.2.3
py-cpuinfo==9.0.0
pyairports==2.1.1
pyarrow==19.0.0
pybind11==2.13.6
pycountry==24.6.1
pydantic==2.10.6
pydantic_core==2.27.2
Pygments==2.19.1
pytest==8.3.4
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-multipart==0.0.20
pytz==2025.1
PyYAML==6.0.2
pyzmq==26.2.1
ray==2.42.0
referencing==0.36.2
regex==2024.11.6
requests==2.32.3
rpds-py==0.22.3
safetensors==0.5.2
sentencepiece==0.2.0
setproctitle==1.3.4
sgl-kernel==0.0.3.post1
sglang==0.4.2.post2
six==1.17.0
sniffio==1.3.1
stack-data==0.6.3
starlette==0.45.3
sympy==1.13.1
tiktoken==0.7.0
tokenizers==0.21.0
tomli==2.2.1
torch==2.5.1
torchao==0.8.0
torchvision==0.20.1
tqdm==4.67.1
traitlets==5.14.3
transformers==4.48.2
triton==3.1.0
typing_extensions==4.12.2
tzdata==2025.1
urllib3==2.3.0
uvicorn==0.34.0
uvloop==0.21.0
vllm==0.6.4.post1
watchfiles==1.0.4
wcwidth==0.2.13
websockets==14.2
xformers==0.0.28.post3
xgrammar==0.1.11
xxhash==3.5.0
yarl==1.18.3
zipp==3.21.0

serve setup

# node 1
python -m sglang.launch_server --model-path  $MODEL_PATH --tp 16 --dist-init-addr ${IB_IP}:5000 --nnodes 2 --node-rank 0 --trust-remote-code --served-model-name dpsk-r1 --host 0.0.0.0
 
# node 2
python -m sglang.launch_server --model-path $MODEL_PATH --tp 16 --dist-init-addr ${IB_IP}:5000 --nnodes 2 --node-rank 1 --trust-remote-code --served-model-name dpsk-r1 --host 0.0.0.0

benchmark

Image

Error Info

on node 1

2025-02-09 10:14:05 TP0] Decode batch. #running-req: 2, #token: 3698, token usage: 0.01, gen throughput (token/s): 50.46, #queue-req: 0
[2025-02-09 10:14:06 TP0] Decode batch. #running-req: 2, #token: 3778, token usage: 0.01, gen throughput (token/s): 50.47, #queue-req: 0
[2025-02-09 10:14:08 TP0] Decode batch. #running-req: 2, #token: 3858, token usage: 0.01, gen throughput (token/s): 50.41, #queue-req: 0
[2025-02-09 10:14:09 TP0] Decode batch. #running-req: 2, #token: 3938, token usage: 0.01, gen throughput (token/s): 50.34, #queue-req: 0
[2025-02-09 10:14:11] INFO:     10.39.13.144:48590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-02-09 10:14:11 TP0] Decode batch. #running-req: 1, #token: 2162, token usage: 0.01, gen throughput (token/s): 46.21, #queue-req: 0
[2025-02-09 10:14:12] INFO:     10.39.13.144:48606 - "POST /v1/chat/completions HTTP/1.1" 200 OK

[2025-02-09 11:06:40 TP0] Prefill batch. #new-seq: 2, #new-token: 632, #cached-token: 126, cache hit rate: 80.75%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2025-02-09 11:06:41 TP0] Prefill batch. #new-seq: 3, #new-token: 170, #cached-token: 837, cache hit rate: 80.76%, token usage: 0.00, #running-req: 2, #queue-req: 0
[2025-02-09 11:06:41 TP0] Prefill batch. #new-seq: 2, #new-token: 359, #cached-token: 558, cache hit rate: 80.68%, token usage: 0.00, #running-req: 5, #queue-req: 0
[2025-02-09 11:06:42 TP0] Decode batch. #running-req: 7, #token: 1029, token usage: 0.00, gen throughput (token/s): 0.02, #queue-req: 0
[2025-02-09 11:06:42 TP0] Prefill batch. #new-seq: 1, #new-token: 76, #cached-token: 279, cache hit rate: 80.67%, token usage: 0.00, #running-req: 7, #queue-req: 0
[2025-02-09 11:06:42 TP0] Prefill batch. #new-seq: 1, #new-token: 15, #cached-token: 279, cache hit rate: 80.69%, token usage: 0.00, #running-req: 8, #queue-req: 0
[2025-02-09 11:06:42 TP0] Prefill batch. #new-seq: 4, #new-token: 128, #cached-token: 1116, cache hit rate: 80.74%, token usage: 0.00, #running-req: 9, #queue-req: 0
[2025-02-09 11:06:43 TP0] Prefill batch. #new-seq: 18, #new-token: 1652, #cached-token: 5022, cache hit rate: 80.58%, token usage: 0.00, #running-req: 13, #queue-req: 0
[2025-02-09 11:06:43 TP0] Prefill batch. #new-seq: 52, #new-token: 3359, #cached-token: 14512, cache hit rate: 80.63%, token usage: 0.01, #running-req: 31, #queue-req: 2
[2025-02-09 11:13:23 TP4] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:24 TP5] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:24 TP2] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:24 TP6] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:24 TP1] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:24 TP3] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:24 TP7] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:25 TP0] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:28] Received sigquit from a child proces. It usually means the child failed.
[1]    25183 killed     python -m sglang.launch_server --model-path  --tp 16 --dist-init-addr   2  0 

on node 2

[2025-02-09 07:00:58 TP12] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /cpfs/user/liuyanjiang/miniconda3/envs/sglang/lib/python3.10/site-packages/sglang/srt/layers/quantization/configs/N=2048,K=512,device_name=NVIDIA_L20Z,dtype=fp8_w8a8,block_shape=[128, 128].json
[2025-02-09 07:00:58 TP8] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /cpfs/user/liuyanjiang/miniconda3/envs/sglang/lib/python3.10/site-packages/sglang/srt/layers/quantization/configs/N=2048,K=512,device_name=NVIDIA_L20Z,dtype=fp8_w8a8,block_shape=[128, 128].json
[2025-02-09 07:00:58 TP9] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /cpfs/user/liuyanjiang/miniconda3/envs/sglang/lib/python3.10/site-packages/sglang/srt/layers/quantization/configs/N=2048,K=512,device_name=NVIDIA_L20Z,dtype=fp8_w8a8,block_shape=[128, 128].json
[2025-02-09 11:13:23 TP9] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:23 TP11] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:23 TP15] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:24 TP8] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:24 TP12] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:24 TP13] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:24 TP14] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:25 TP10] Watchdog timeout (self.watchdog_timeout=300)
[2025-02-09 11:13:28] Received sigquit from a child proces. It usually means the child failed.
[1]    21769 killed     python -m sglang.launch_server --model-path  --tp 16 --dist-init-addr   2  1 

system monitor

both nodes’ gpu memory usage is round 75%, not even the crash moment node 1 gpu utilization

Image

node 2 gpu utilization

Image

occurrence frequency

0.5-4hours when requests any parallel degree of 32-128

Environment

Python: 3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA L20Z GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.5, V12.5.82 CUDA Driver Version: 550.127.08 PyTorch: 2.5.1+cu124 sglang: 0.4.2.post2 flashinfer: 0.2.0.post2+cu124torch2.5 triton: 3.1.0 transformers: 4.48.2 torchao: 0.8.0 numpy: 1.26.4 aiohttp: 3.11.12 fastapi: 0.115.8 hf_transfer: 0.1.9 huggingface_hub: 0.28.1 interegular: 0.3.3 modelscope: 1.22.3 orjson: 3.10.15 packaging: 24.2 psutil: 6.1.1 pydantic: 2.10.6 multipart: 0.0.20 zmq: 26.2.1 uvicorn: 0.34.0 uvloop: 0.21.0 vllm: 0.6.4.post1 openai: 1.61.1 anthropic: 0.45.2 decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-127 0-1 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-127 0-1 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-127 0-1 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-127 0-1 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-127 0-1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-127 0-1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-127 0-1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X PHB PHB PHB PHB PHB PHB PHB PHB 0-127 0-1 N/A NIC0 PHB PHB PHB PHB PHB PHB PHB PHB X PHB PHB PHB PHB PHB PHB PHB NIC1 PHB PHB PHB PHB PHB PHB PHB PHB PHB X PHB PHB PHB PHB PHB PHB NIC2 PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB X PHB PHB PHB PHB PHB NIC3 PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB X PHB PHB PHB PHB NIC4 PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB X PHB PHB PHB NIC5 PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB X PHB PHB NIC6 PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB X PHB NIC7 PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB X

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7

Hypervisor vendor: KVM ulimit soft: 102400

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 4
  • Comments: 17 (5 by maintainers)

Most upvoted comments

#3709 should fix this bug. Let me know if you encounter an issue after applying it.

Watchdog timeout (self.watchdog_timeout=300) +1

+1