sglang: [Bug] Tensor model parallel group is not initialized when deploying Qwen3-30B-A3B-AWQ
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
Hi, SGLang Team,
I am using it to deploy an AWQ quantized model of Qwen3-30B-A3B: swift/Qwen3-30B-A3B-AWQ from modelscope. but encounter the following issue:
File "/home/a/sglang/python/sglang/srt/managers/scheduler.py", line 2215, in run_scheduler_process
scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, pp_rank, dp_rank)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/a/sglang/python/sglang/srt/managers/scheduler.py", line 268, in __init__
self.tp_worker = TpWorkerClass(
^^^^^^^^^^^^^^
File "/home/a/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 64, in __init__
self.worker = TpModelWorker(
^^^^^^^^^^^^^^
File "/home/a/sglang/python/sglang/srt/managers/tp_worker.py", line 81, in __init__
self.model_runner = ModelRunner(
^^^^^^^^^^^^
File "/home/a/sglang/python/sglang/srt/model_executor/model_runner.py", line 190, in __init__
self.initialize(min_per_gpu_memory)
File "/home/a/sglang/python/sglang/srt/model_executor/model_runner.py", line 205, in initialize
self.load_model()
File "/home/a/sglang/python/sglang/srt/model_executor/model_runner.py", line 458, in load_model
self.model = get_model(
^^^^^^^^^^
File "/home/a/sglang/python/sglang/srt/model_loader/__init__.py", line 22, in get_model
return loader.load_model(
^^^^^^^^^^^^^^^^^^
File "/home/a/sglang/python/sglang/srt/model_loader/loader.py", line 377, in load_model
model.load_weights(self._get_all_weights(model_config, model))
File "/home/a/sglang/python/sglang/srt/models/qwen3_moe.py", line 406, in load_weights
weight_loader(
File "/data2/a/miniconda3/envs/cap/lib/python3.11/site-packages/vllm/model_executor/layers/quantization/moe_wna16.py", line 396, in moe_wna16_weight_loader
device = get_tp_group().device
^^^^^^^^^^^^^^
File "/data2/a/miniconda3/envs/cap/lib/python3.11/site-packages/vllm/distributed/parallel_state.py", line 749, in get_tp_group
assert _TP is not None, ("tensor model parallel group is not initialized")
^^^^^^^^^^^^^^^
AssertionError: tensor model parallel group is not initialized
Reproduction
Running Command:
modelscope download swift/Qwen3-30B-A3B-AWQ
python3 -m sglang.launch_server --model-path /home/a/.cache/modelscope/hub/models/swift/Qwen3-30B-A3B-AWQ --attention-backend flashinfer --mem-fraction-static 0.9
Environment
Sglang 0.4.6.post2, 8xH20
Python: 3.11.11 (main, Dec 11 2024, 16:28:39) [GCC 11.2.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA H20 GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.4, V12.4.99 CUDA Driver Version: 535.161.07 PyTorch: 2.6.0+cu124 sglang: 0.4.6.post2 sgl_kernel: 0.1.1 flashinfer_python: 0.2.5 triton: 3.2.0 transformers: 4.51.1 torchao: 0.10.0 numpy: 2.2.5 aiohttp: 3.11.18 fastapi: 0.115.12 hf_transfer: 0.1.9 huggingface_hub: 0.30.2 interegular: 0.3.3 modelscope: 1.25.0 orjson: 3.10.18 outlines: 0.1.11 packaging: 25.0 psutil: 7.0.0 pydantic: 2.11.4 python-multipart: 0.0.20 pyzmq: 26.4.0 uvicorn: 0.34.2 uvloop: 0.21.0 vllm: 0.8.4 xgrammar: 0.1.18 openai: 1.75.0 tiktoken: 0.9.0 anthropic: 0.50.0 litellm: 1.67.6 decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE NODE SYS SYS SYS SYS 0-95,192-287 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX PHB NODE SYS SYS SYS SYS 0-95,192-287 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE PHB PIX NODE SYS SYS SYS SYS 0-95,192-287 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE PIX SYS SYS SYS SYS 0-95,192-287 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE NODE NODE 96-191,288-383 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX NODE NODE 96-191,288-383 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS NODE NODE PIX PHB 96-191,288-383 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS NODE NODE PHB PIX 96-191,288-383 1 N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS NIC1 NODE PIX PHB NODE SYS SYS SYS SYS NODE X PHB NODE SYS SYS SYS SYS NIC2 NODE PHB PIX NODE SYS SYS SYS SYS NODE PHB X NODE SYS SYS SYS SYS NIC3 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X SYS SYS SYS SYS NIC4 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE NIC5 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE NIC6 SYS SYS SYS SYS NODE NODE PIX PHB SYS SYS SYS SYS NODE NODE X PHB NIC7 SYS SYS SYS SYS NODE NODE PHB PIX SYS SYS SYS SYS NODE NODE PHB X
Legend:
X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_bond_0 NIC1: mlx5_bond_1 NIC2: mlx5_bond_2 NIC3: mlx5_bond_3 NIC4: mlx5_bond_4 NIC5: mlx5_bond_5 NIC6: mlx5_bond_6 NIC7: mlx5_bond_7
ulimit soft: 1000000
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18
try –quantization=awq_marlin
Hi, I resolve this issue in this PR, but setting
quantization=awq_marlinis necessary. To be honest the issue is still happening when trying to load this model with TP. This might be an issue related to vLLM or something in the middle.