sglang: [Bug] common_ops.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

Hello,

I successfully built the sgl-kernel with sm_120 (NVIDIA RTX 50 series) and CUDA 12.8, but encountered the following issue when running sglang.launch_server command. Please help.

INFO 04-06 14:51:04 [__init__.py:256] Automatically detected platform cuda.
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/admin2/Projects/sglang/python/sglang/launch_server.py", line 6, in <module>
    from sglang.srt.entrypoints.http_server import launch_server
  File "/home/admin2/Projects/sglang/python/sglang/srt/entrypoints/http_server.py", line 45, in <module>
    from sglang.srt.entrypoints.engine import _launch_subprocesses
  File "/home/admin2/Projects/sglang/python/sglang/srt/entrypoints/engine.py", line 40, in <module>
    from sglang.srt.managers.data_parallel_controller import (
  File "/home/admin2/Projects/sglang/python/sglang/srt/managers/data_parallel_controller.py", line 27, in <module>
    from sglang.srt.managers.io_struct import (
  File "/home/admin2/Projects/sglang/python/sglang/srt/managers/io_struct.py", line 25, in <module>
    from sglang.srt.managers.schedule_batch import BaseFinishReason
  File "/home/admin2/Projects/sglang/python/sglang/srt/managers/schedule_batch.py", line 45, in <module>
    from sglang.srt.configs.model_config import ModelConfig
  File "/home/admin2/Projects/sglang/python/sglang/srt/configs/model_config.py", line 25, in <module>
    from sglang.srt.layers.quantization import QUANTIZATION_METHODS
  File "/home/admin2/Projects/sglang/python/sglang/srt/layers/quantization/__init__.py", line 54, in <module>
    from sglang.srt.layers.quantization.awq import AWQConfig
  File "/home/admin2/Projects/sglang/python/sglang/srt/layers/quantization/awq.py", line 6, in <module>
    from sgl_kernel import awq_dequantize
  File "/home/admin2/.virtualenvs/sglang/lib/python3.10/site-packages/sgl_kernel/__init__.py", line 12, in <module>
    from sgl_kernel import common_ops
ImportError: /home/admin2/.virtualenvs/sglang/lib/python3.10/site-packages/sgl_kernel/common_ops.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb

Reproduction

  
  python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --dtype bfloat16 \
    --attention-backend flashinfer \
    --enable-torch-compile 

Environment

Python: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0] CUDA available: True GPU 0: NVIDIA GeForce RTX 5090 GPU 0 Compute Capability: 12.0 CUDA_HOME: /usr/local/cuda-12.8 NVCC: Cuda compilation tools, release 12.8, V12.8.93 CUDA Driver Version: 570.124.06 PyTorch: 2.8.0.dev20250405+cu128 sglang: 0.4.4.post4 sgl_kernel: 0.0.8 flashinfer: Module Not Found triton: 3.3.0 transformers: 4.51.0 torchao: 0.9.0 numpy: 1.26.4 aiohttp: 3.11.14 fastapi: 0.115.11 hf_transfer: 0.1.9 huggingface_hub: 0.30.1 interegular: 0.3.3 modelscope: 1.24.0 orjson: 3.10.15 outlines: 0.1.11 packaging: 24.2 psutil: 7.0.0 pydantic: 2.10.6 multipart: Module Not Found zmq: Module Not Found uvicorn: 0.34.0 uvloop: 0.21.0 vllm: 0.7.4.dev254+ged6ea065.d20250319.cu128 xgrammar: 0.1.11 openai: 1.67.0 tiktoken: 0.9.0 anthropic: 0.49.0 litellm: 1.63.12 decord: 0.6.0 NVIDIA Topology: GPU0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X 0-63 0 N/A

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

ulimit soft: 1024

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 32 (3 by maintainers)

Most upvoted comments

@shahizat Hi, maybe you need to try use torch 2.5.1 for prebuild whl, or try make build to build a whl for yourself. sgl-kernel prebuild whl need specific torch version.

The environment I’m using is torch 2.5.1 + cu118. After running make build, I still encountered the same issue.

ImportError: env/lib/python3.10/site-packages/sgl_kernel/common_ops.abi3.so: undefined symbol: _Z24ApplyTokenBitmaskInplaceN2at6TensorES0_St8optionalIS0_E

How to solve this?

Same issue here. How to solve?

@shahizat Hi, maybe you need to try use torch 2.5.1 for prebuild whl, or try make build to build a whl for yourself. sgl-kernel prebuild whl need specific torch version.

The environment I’m using is torch 2.5.1 + cu118. After running make build, I still encountered the same issue.

ImportError: env/lib/python3.10/site-packages/sgl_kernel/common_ops.abi3.so: undefined symbol: _Z24ApplyTokenBitmaskInplaceN2at6TensorES0_St8optionalIS0_E

How to solve this?

+1

Hi all, I am working on fix cu118 version. It will fix these days.

Hi, I tried the latest sgl-kernel-0.1.3, but still have this problem. my cuda version is 12.1

@toncao It should be fixed in the latest version

it seems to be a problem of sql_kernel, after reading https://github.com/modal-labs/modal-examples/pull/1149#issuecomment-2845895135 I bisected to sgl-kernel 0.1.0 to 0.1.1 upgrade. Seems to work now by pinning to old 0.1.0. 😮‍💨

Not sure what changed exactly.

got the same problem after the installation following the install instruction myenv: cuda-12.6, torch-2.5.1-cu124, sglang-0.4.3.post4, flashinfer-python 0.2.5, sgl-kernel-0.1.2-post1 Should I downgrade or upgrade sgl-kernel to fix this problem?

_ZNK3c1011StorageImpl27throw_data_ptr_access_errorEv

hi, I meet the same problem:“ZNK3c1011StorageImpl27throw_data_ptr_access_errorEv”, have you solved?

got the same problem after the installation following the install instruction myenv: cuda-12.6, torch-2.5.1-cu124, sglang-0.4.3.post4, flashinfer-python 0.2.5, sgl-kernel-0.1.2-post1 Should I downgrade or upgrade sgl-kernel to fix this problem?

_ZNK3c1011StorageImpl27throw_data_ptr_access_errorEv

+1. sglang 0.4.6.post3, nvidia driver 575.51.02, cuda_12.9 also have this problem. Changing attention backend to flashinfer does not help.

@shahizat Hi, maybe you need to try use torch 2.5.1 for prebuild whl, or try make build to build a whl for yourself. sgl-kernel prebuild whl need specific torch version.

The environment I’m using is torch 2.5.1 + cu118. After running make build, I still encountered the same issue.

ImportError: env/lib/python3.10/site-packages/sgl_kernel/common_ops.abi3.so: undefined symbol: _Z24ApplyTokenBitmaskInplaceN2at6TensorES0_St8optionalIS0_E

How to solve this?

I believe this is a bug in torch/triton, not caused by SGLang. It appears during torch.compile when inductor processes the generated triton kernel. This might be an issue with the torch/triton version.

@shahizat Hi, maybe you need to try use torch 2.5.1 for prebuild whl, or try make build to build a whl for yourself. sgl-kernel prebuild whl need specific torch version.