sglang: [Bug] Some FP8 models fail to load

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

Sglang crashes when loading some FP8 models. I only got Qwen3 FP8 model work, but not others from RedHatAI: WORKS: https://huggingface.co/Qwen/Qwen3-32B-FP8

DO NOT WORK https://huggingface.co/RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic https://huggingface.co/RedHatAI/Qwen3-32B-FP8-dynamic https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic https://huggingface.co/RedHatAI/gemma-3-27b-it-FP8-dynamic

Reproduction

python3.12 -m venv .venv
source .venv/bin/activate.fish
python -m pip install --upgrade pip
python -m pip install uv
python -m uv pip install "sglang[all]>=0.4.7.post1"
python -m uv pip install -U torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
python -m uv pip install --force-reinstall sgl_kernel-0.1.9+cu128-cp39-abi3-manylinux2014_x86_64.whl
python -m uv pip install -U nvidia-nccl-cu12

WORKS with Qwen3-4B normal fp16 -tp 1 CUDA_VISIBLE_DEVICES=2 python -m sglang.launch_server --model-path /mnt/llms/models/Qwen/Qwen3-4B --tp 1 --context-length 2048 --port 5001 --host 0.0.0.0

WORKS with Qwen3-4B-FP8 -tp 1 (I have to add SGL_ENABLE_JIT_DEEPGEMM=0) SGL_ENABLE_JIT_DEEPGEMM=0 CUDA_VISIBLE_DEVICES=2 python -m sglang.launch_server --model-path /mnt/llms/models/Qwen/Qwen3-4B-FP8/ --tp 1 --context-length 2048 --port 5001 --host 0.0.0.0

DOES NOT Work with RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic SGL_ENABLE_JIT_DEEPGEMM=0 CUDA_VISIBLE_DEVICES=2 python -m sglang.launch_server --model-path /mnt/llms/models/RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic/ --tp 1 --context-length 2048 --port 5001 --host 0.0.0.0

Environment

 python3 -m sglang.check_env
Python: 3.12.11 (main, Jun  4 2025, 08:56:18) [GCC 11.4.0]
CUDA available: True
GPU 0,1,3,5: NVIDIA GeForce RTX 3090
GPU 2,4: NVIDIA GeForce RTX 5090
GPU 0,1,3,5 Compute Capability: 8.6
GPU 2,4 Compute Capability: 12.0
CUDA_HOME: /usr/local/cuda-12.8
NVCC: Cuda compilation tools, release 12.8, V12.8.93
CUDA Driver Version: 575.57.08
PyTorch: 2.7.1+cu128
sglang: 0.4.7.post1
sgl_kernel: 0.1.9
flashinfer_python: 0.2.6.post1
triton: 3.3.1
transformers: 4.52.3
torchao: 0.9.0
numpy: 2.3.1
aiohttp: 3.12.13
fastapi: 0.115.13
hf_transfer: 0.1.9
huggingface_hub: 0.33.0
interegular: 0.3.3
modelscope: 1.27.1
orjson: 3.10.18
outlines: 0.1.11
packaging: 25.0
psutil: 7.0.0
pydantic: 2.11.7
python-multipart: 0.0.20
pyzmq: 27.0.0
uvicorn: 0.34.3
uvloop: 0.21.0
vllm: Module Not Found
xgrammar: 0.1.19
openai: 1.91.0
tiktoken: 0.9.0
anthropic: 0.55.0
litellm: 1.73.0
decord: 0.6.0
NVIDIA Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    NODE    NODE    NODE    NODE    0-47    0               N/A
GPU1    NODE     X      NODE    NODE    NODE    NODE    0-47    0               N/A
GPU2    NODE    NODE     X      PHB     NODE    NODE    0-47    0               N/A
GPU3    NODE    NODE    PHB      X      NODE    NODE    0-47    0               N/A
GPU4    NODE    NODE    NODE    NODE     X      PHB     0-47    0               N/A
GPU5    NODE    NODE    NODE    NODE    PHB      X      0-47    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

ulimit soft: 1024

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Reactions: 1
  • Comments: 16

Most upvoted comments

I will look into the problem later this week

Does this error still occur when you disable the CUDA graph?

No, a different error:

CUDA_VISIBLE_DEVICES=4 python -m sglang.launch_server --model-path /mnt/llms/models/RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic/ --tp 1 --context-length 2048 --port 5001 --host 0.0.0.0 --disable-cuda-graph
[2025-07-01 15:50:37] server_args=ServerArgs(model_path='/mnt/llms/models/RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic/', tokenizer_path='/mnt/llms/models/RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic/', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=2048, device='cuda', served_model_name='/mnt/llms/models/RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic/', chat_template=None, completion_template=None, is_embedding=False, enable_multimodal=None, revision=None, impl='auto', host='0.0.0.0', port=5001, mem_fraction_static=0.879, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=2048, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=247102948, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, bucket_time_to_first_token=None, bucket_e2e_request_latency=None, bucket_inter_token_latency=None, collect_tokens_histogram=False, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, ep_size=1, enable_ep_moe=False, enable_deepep_moe=False, enable_flashinfer_moe=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, cuda_graph_max_bs=8, cuda_graph_bs=None, disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_nccl_nvls=False, enable_tokenizer_batch_encode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, disable_overlap_schedule=False, disable_overlap_cg_plan=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_torch_compile=False, torch_compile_max_bs=32, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, enable_return_hidden_states=False, warmups=None, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, debug_tensor_dump_prefill_only=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, num_reserved_decode_tokens=512, pdlb_url=None, custom_weight_loader=[], weight_loader_disable_mmap=False)
[2025-07-01 15:50:44] Attention backend not set. Use flashinfer backend by default.
[2025-07-01 15:50:44] Init torch distributed begin.
[W701 15:50:45.905511580 socket.cpp:755] [c10d] The client socket cannot be initialized to connect to [localhost]:5630 (errno: 97 - Address family not supported by protocol).
[2025-07-01 15:50:45] Init torch distributed ends. mem usage=0.00 GB
[2025-07-01 15:50:45] Ignore import error when loading sglang.srt.models.gemma3n_audio. cannot import name 'Gemma3nAudioConfig' from 'transformers' (/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/transformers/__init__.py)
[2025-07-01 15:50:45] Ignore import error when loading sglang.srt.models.gemma3n_causal. cannot import name 'Gemma3nTextConfig' from 'transformers' (/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/transformers/__init__.py)
[2025-07-01 15:50:45] Ignore import error when loading sglang.srt.models.gemma3n_mm. cannot import name 'Gemma3nAudioConfig' from 'transformers' (/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/transformers/__init__.py)
[2025-07-01 15:50:45] Load weight begin. avail mem=30.73 GB
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  1.19it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.13it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.14it/s]

[2025-07-01 15:50:48] Load weight end. type=LlamaForCausalLM, dtype=torch.bfloat16, avail mem=22.19 GB, mem usage=8.54 GB.
[2025-07-01 15:50:48] KV Cache is allocated. #tokens: 151244, K size: 9.23 GB, V size: 9.23 GB
[2025-07-01 15:50:48] Memory pool end. avail mem=3.59 GB
[2025-07-01 15:50:49] max_total_num_tokens=151244, chunked_prefill_size=2048, max_prefill_tokens=16384, max_running_requests=4096, context_len=2048, available_gpu_mem=3.10 GB
[2025-07-01 15:50:49] INFO:     Started server process [3357293]
[2025-07-01 15:50:49] INFO:     Waiting for application startup.
[2025-07-01 15:50:49] INFO:     Application startup complete.
[2025-07-01 15:50:49] INFO:     Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)
[2025-07-01 15:50:50] INFO:     127.0.0.1:55308 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-07-01 15:50:50] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0
[2025-07-01 15:50:51] TpModelWorkerClient hit an exception: Traceback (most recent call last):
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 127, in forward_thread_func
    self.forward_thread_func_()
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 162, in forward_thread_func_
    self.worker.forward_batch_generation(
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 212, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 1231, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 1260, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 1199, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/models/llama.py", line 457, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/models/llama.py", line 334, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/models/llama.py", line 258, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/models/llama.py", line 186, in forward
    qkv, _ = self.qkv_proj(hidden_states)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/linear.py", line 462, in forward
    output_parallel = self.quant_method.apply(self, input_, bias)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 664, in apply
    return scheme.apply_weights(layer, x, bias=bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py", line 152, in apply_weights
    return apply_fp8_linear(
           ^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/fp8_utils.py", line 561, in apply_fp8_linear
    output = fp8_scaled_mm(
             ^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sgl_kernel/gemm.py", line 35, in fp8_scaled_mm
    return torch.ops.sgl_kernel.fp8_scaled_mm.default(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/_ops.py", line 756, in __call__
    return self._op(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected status == cutlass::Status::kSuccess to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)

[2025-07-01 15:50:51] Received sigquit from a child process. It usually means the child failed.
fish: Job 1, 'CUDA_VISIBLE_DEVICES=4 python -…' terminated by signal SIGKILL (Forced quit)

Could you please offer the specific hardware and env you are using? I can not reproduce the error on H20+cuda12.8

Running sgl-kernel/tests/test_fp8_gemm.py on the 5090 will trigger the issue, and it eventually hangs inside the torch.ops.sgl_kernel.fp8_scaled_mm.default function.

I do not have access to 5090 for now. Would you mind updating PyTorch to 2.8.0 with sm120 support and compiling the SGL kernel from source with CUDA 12.8, which might solve the problem?

I’ll give it a try now — I’ll get back to you in a moment.

The latest PyTorch version is 2.7.1 — do I need to install the nightly build instead?

I was able to compile sglang using the nightly version of torch without any issues, but vLLM doesn’t work. Is there an alternative to this operator?

>>> import torch
tor>>> torch.ops
<module 'torch.ops' from '_ops.py'>
>>> torch.ops.sgl_kernel
<module 'torch.ops.sgl_kernel' from 'torch.ops'>
>>> torch.ops.sgl_kernel.fp8_scaled_mm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/anaconda3/lib/python3.11/site-packages/torch/_ops.py", line 1353, in __getattr__
    raise AttributeError(
AttributeError: '_OpNamespace' 'sgl_kernel' object has no attribute 'fp8_scaled_mm'
>>> torch.__version__
'2.9.0.dev20250701+cu128'

Also, this operator cannot be found inside sgl_kernel.

we are working on removing the vLLM dependency of quantization. that might help resolve the vLLM issue.

wondering if this commit fixed fp8 a1c1ebe

no, its not working. I updated everything to yesterdays release, version sglang 0.4.8 and sgl_kernel 0.2.0:

python3.12 -m venv .venv
source .venv/bin/activate.fish
python -m pip install --upgrade pip
python -m pip install uv
python -m uv pip install "sglang[all]>=0.4.8"
python -m uv pip install --force-reinstall sgl_kernel-0.2.0+cu128-cp39-abi3-manylinux2014_x86_64.whl
python -m uv pip install -U torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
python -m uv pip install -U nvidia-nccl-cu12

SGL_ENABLE_JIT_DEEPGEMM=0 CUDA_VISIBLE_DEVICES=2,4 python -m sglang.launch_server --model-path /mnt/llms/models/RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic/ --tp 2 --context-length 2048 --port 5001 --host 0.0.0.0

Output error:

[2025-06-25 21:32:23] server_args=ServerArgs(model_path='/mnt/llms/models/RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic/', tokenizer_path='/mnt/llms/models/RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic/', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=2048, device='cuda', served_model_name='/mnt/llms/models/RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic/', chat_template=None, completion_template=None, is_embedding=False, enable_multimodal=None, revision=None, impl='auto', host='0.0.0.0', port=5001, mem_fraction_static=0.875, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=2048, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=2, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=817694184, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, bucket_time_to_first_token=None, bucket_e2e_request_latency=None, bucket_inter_token_latency=None, collect_tokens_histogram=False, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, ep_size=1, enable_ep_moe=False, enable_deepep_moe=False, enable_flashinfer_moe=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, cuda_graph_max_bs=8, cuda_graph_bs=None, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_nccl_nvls=False, enable_tokenizer_batch_encode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, disable_overlap_schedule=False, disable_overlap_cg_plan=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_torch_compile=False, torch_compile_max_bs=32, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, enable_return_hidden_states=False, warmups=None, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, debug_tensor_dump_prefill_only=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, num_reserved_decode_tokens=512, pdlb_url=None, custom_weight_loader=[], weight_loader_disable_mmap=False)
[2025-06-25 21:32:30 TP0] Attention backend not set. Use flashinfer backend by default.
[2025-06-25 21:32:30 TP0] Init torch distributed begin.
[W625 21:32:30.371561343 socket.cpp:755] [c10d] The client socket cannot be initialized to connect to [localhost]:5736 (errno: 97 - Address family not supported by protocol).
[W625 21:32:30.387369400 socket.cpp:755] [c10d] The client socket cannot be initialized to connect to [localhost]:5736 (errno: 97 - Address family not supported by protocol).
[2025-06-25 21:32:30 TP0] sglang is using nccl==2.27.3
[2025-06-25 21:32:30 TP1] Setup Custom allreduce failed with CUDART error: peer access is not supported between these two devices. To silence this warning, specify --disable-custom-all-reduce explicitly.
[2025-06-25 21:32:30 TP0] Setup Custom allreduce failed with CUDART error: peer access is not supported between these two devices. To silence this warning, specify --disable-custom-all-reduce explicitly.
[2025-06-25 21:32:30 TP0] Init torch distributed ends. mem usage=0.28 GB
[2025-06-25 21:32:31 TP0] Load weight begin. avail mem=30.58 GB
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  1.58it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.64it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.63it/s]

[2025-06-25 21:32:33 TP0] Load weight end. type=LlamaForCausalLM, dtype=torch.bfloat16, avail mem=26.26 GB, mem usage=4.32 GB.
[2025-06-25 21:32:33 TP1] KV Cache is allocated. #tokens: 367128, K size: 11.20 GB, V size: 11.20 GB
[2025-06-25 21:32:33 TP0] KV Cache is allocated. #tokens: 367128, K size: 11.20 GB, V size: 11.20 GB
[2025-06-25 21:32:33 TP0] Memory pool end. avail mem=3.73 GB
[2025-06-25 21:32:33 TP0] Capture cuda graph begin. This can take up to several minutes. avail mem=3.24 GB
Capture cuda graph bs [1, 2, 4, 8]
Capturing batches (avail_mem=3.24 GB):   0%|                                                                                                           | 0/4 [00:00<?, ?it/s]
[2025-06-25 21:32:34 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 336, in __init__
    self.capture()
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 436, in capture
    ) = self.capture_one_batch_size(bs, forward)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 587, in capture_one_batch_size
    run_once()
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 575, in run_once
    logits_output_or_pp_proxy_tensors = forward(
                                        ^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/models/llama.py", line 457, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/models/llama.py", line 334, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/models/llama.py", line 258, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/models/llama.py", line 186, in forward
    qkv, _ = self.qkv_proj(hidden_states)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/linear.py", line 445, in forward
    output_parallel = self.quant_method.apply(self, input_, bias)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 664, in apply
    return scheme.apply_weights(layer, x, bias=bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py", line 152, in apply_weights
    return apply_fp8_linear(
           ^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/fp8_utils.py", line 560, in apply_fp8_linear
    output = fp8_scaled_mm(
             ^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sgl_kernel/gemm.py", line 35, in fp8_scaled_mm
    return torch.ops.sgl_kernel.fp8_scaled_mm.default(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/_ops.py", line 756, in __call__
    return self._op(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected status == cutlass::Status::kSuccess to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2631, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, pp_rank, dp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 315, in __init__
    self.tp_worker = TpWorkerClass(
                     ^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 64, in __init__
    self.worker = TpModelWorker(
                  ^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 79, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 220, in __init__
    self.initialize(min_per_gpu_memory)
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 300, in initialize
    self.init_cuda_graphs()
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 1154, in init_cuda_graphs
    self.cuda_graph_runner = CudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 338, in __init__
    raise Exception(
Exception: Capture cuda graph failed: Expected status == cutlass::Status::kSuccess to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
Possible solutions:
1. set --mem-fraction-static to a smaller value (e.g., 0.8 or 0.7)
2. set --cuda-graph-max-bs to a smaller value (e.g., 16)
3. disable torch compile by not using --enable-torch-compile
4. disable CUDA graph by --disable-cuda-graph. (Not recommended. Huge performance loss)
Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose


[2025-06-25 21:32:34 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 336, in __init__
    self.capture()
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 436, in capture
    ) = self.capture_one_batch_size(bs, forward)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 587, in capture_one_batch_size
    run_once()
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 575, in run_once
    logits_output_or_pp_proxy_tensors = forward(
                                        ^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/models/llama.py", line 457, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/models/llama.py", line 334, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/models/llama.py", line 258, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/models/llama.py", line 186, in forward
    qkv, _ = self.qkv_proj(hidden_states)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/linear.py", line 445, in forward
    output_parallel = self.quant_method.apply(self, input_, bias)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 664, in apply
    return scheme.apply_weights(layer, x, bias=bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py", line 152, in apply_weights
    return apply_fp8_linear(
           ^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/fp8_utils.py", line 560, in apply_fp8_linear
    output = fp8_scaled_mm(
             ^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sgl_kernel/gemm.py", line 35, in fp8_scaled_mm
    return torch.ops.sgl_kernel.fp8_scaled_mm.default(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/_ops.py", line 756, in __call__
    return self._op(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected status == cutlass::Status::kSuccess to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2631, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, pp_rank, dp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 315, in __init__
    self.tp_worker = TpWorkerClass(
                     ^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 64, in __init__
    self.worker = TpModelWorker(
                  ^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 79, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 220, in __init__
    self.initialize(min_per_gpu_memory)
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 300, in initialize
    self.init_cuda_graphs()
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 1154, in init_cuda_graphs
    self.cuda_graph_runner = CudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 338, in __init__
    raise Exception(
Exception: Capture cuda graph failed: Expected status == cutlass::Status::kSuccess to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
Possible solutions:
1. set --mem-fraction-static to a smaller value (e.g., 0.8 or 0.7)
2. set --cuda-graph-max-bs to a smaller value (e.g., 16)
3. disable torch compile by not using --enable-torch-compile
4. disable CUDA graph by --disable-cuda-graph. (Not recommended. Huge performance loss)
Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose


[2025-06-25 21:32:34] Received sigquit from a child process. It usually means the child failed.
fish: Job 1, 'SGL_ENABLE_JIT_DEEPGEMM=0 CUDA_…' terminated by signal SIGKILL (Forced quit)