sglang: [Bug] maximum recursion depth exceeded
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
Maximum recursion depth triggered on exception exit.
self._send_signal(sig)
File "/usr/local/lib/python3.10/dist-packages/psutil/__init__.py", line 1266, in _send_signal
os.kill(self.pid, sig)
File "/sgl-workspace/sglang/python/sglang/srt/entrypoints/engine.py", line 333, in sigquit_handler
kill_process_tree(os.getpid())
File "/sgl-workspace/sglang/python/sglang/srt/utils.py", line 492, in kill_process_tree
children = itself.children(recursive=True)
File "/usr/local/lib/python3.10/dist-packages/psutil/__init__.py", line 971, in children
self._raise_if_pid_reused()
File "/usr/local/lib/python3.10/dist-packages/psutil/__init__.py", line 461, in _raise_if_pid_reused
if self._pid_reused or (not self.is_running() and self._pid_reused):
File "/usr/local/lib/python3.10/dist-packages/psutil/__init__.py", line 636, in is_running
self._pid_reused = self != Process(self.pid)
File "/usr/local/lib/python3.10/dist-packages/psutil/__init__.py", line 319, in __init__
self._init(pid)
File "/usr/local/lib/python3.10/dist-packages/psutil/__init__.py", line 355, in _init
self._ident = self._get_ident()
File "/usr/local/lib/python3.10/dist-packages/psutil/__init__.py", line 396, in _get_ident
return (self.pid, self.create_time())
File "/usr/local/lib/python3.10/dist-packages/psutil/__init__.py", line 778, in create_time
self._create_time = self._proc.create_time()
File "/usr/local/lib/python3.10/dist-packages/psutil/_pslinux.py", line 1716, in wrapper
return fun(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/psutil/_pslinux.py", line 1957, in create_time
ctime = float(self._parse_stat_file()['create_time'])
File "/usr/local/lib/python3.10/dist-packages/psutil/_pslinux.py", line 1716, in wrapper
return fun(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/psutil/_common.py", line 508, in wrapper
raise raise_from(err, None)
File "<string>", line 3, in raise_from
File "/usr/local/lib/python3.10/dist-packages/psutil/_common.py", line 506, in wrapper
return fun(self)
File "/usr/local/lib/python3.10/dist-packages/psutil/_pslinux.py", line 1784, in _parse_stat_file
data = bcat("%s/%s/stat" % (self._procfs_path, self.pid))
File "/usr/local/lib/python3.10/dist-packages/psutil/_common.py", line 851, in bcat
return cat(fname, fallback=fallback, _open=open_binary)
File "/usr/local/lib/python3.10/dist-packages/psutil/_common.py", line 839, in cat
with _open(fname) as f:
File "/usr/local/lib/python3.10/dist-packages/psutil/_common.py", line 799, in open_binary
return open(fname, "rb", buffering=FILE_READ_BUFFER_SIZE)
RecursionError: maximum recursion depth exceeded while calling a Python object
Reproduction
N/A
Environment
root@g1805:/sgl-workspace# python3 -m sglang.check_env INFO 02-12 08:49:13 init.py:190] Automatically detected platform cuda. Python: 3.10.12 (main, Jan 17 2025, 14:35:34) [GCC 11.4.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA GeForce RTX 4090 GPU 0,1,2,3,4,5,6,7 Compute Capability: 8.9 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.4, V12.4.131 CUDA Driver Version: 550.78 PyTorch: 2.5.1+cu124 sgl_kernel: 0.0.3.post3 flashinfer: 0.2.0.post2+cu124torch2.5 triton: 3.1.0 transformers: 4.48.3 torchao: 0.8.0 numpy: 1.26.4 aiohttp: 3.11.12 fastapi: 0.115.8 hf_transfer: 0.1.9 huggingface_hub: 0.28.1 interegular: 0.3.3 modelscope: 1.22.3 orjson: 3.10.15 packaging: 24.2 psutil: 6.1.1 pydantic: 2.10.6 multipart: 0.0.20 zmq: 26.2.1 uvicorn: 0.34.0 uvloop: 0.21.0 vllm: 0.7.2 openai: 1.61.1 tiktoken: 0.8.0 anthropic: 0.45.2 decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PHB SYS SYS SYS SYS SYS SYS 0-27,56-83 0 N/A GPU1 PHB X SYS SYS SYS SYS SYS SYS 0-27,56-83 0 N/A GPU2 SYS SYS X PHB SYS SYS SYS SYS 0-27,56-83 0 N/A GPU3 SYS SYS PHB X SYS SYS SYS SYS 0-27,56-83 0 N/A GPU4 SYS SYS SYS SYS X PHB SYS SYS 28-55,84-111 1 N/A GPU5 SYS SYS SYS SYS PHB X SYS SYS 28-55,84-111 1 N/A GPU6 SYS SYS SYS SYS SYS SYS X PHB 28-55,84-111 1 N/A GPU7 SYS SYS SYS SYS SYS SYS PHB X 28-55,84-111 1 N/A
Legend:
X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks
ulimit soft: 65535
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 3
- Comments: 15
@issaccv I am trying to pass the ci and merge it