sglang: [Bug] main pd version Exception: Failed to encode tensor map: 700

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • 5. Please use English, otherwise it will be closed.

Describe the bug

[2025-05-25 19:45:03] [ERROR] Scheduler hit an exception: Traceback (most recent call last):
  File "/usr/local/src/sglang/python/sglang/srt/managers/scheduler.py", line 2346, in run_scheduler_process
    scheduler.event_loop_overlap_disagg_decode()
  File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/src/sglang/python/sglang/srt/disaggregation/decode.py", line 558, in event_loop_overlap_disagg_decode
    self.process_batch_result(tmp_batch, tmp_result)
  File "/usr/local/src/sglang/python/sglang/srt/managers/scheduler.py", line 1626, in process_batch_result
    self.process_batch_result_decode(batch, result, launch_done)
  File "/usr/local/src/sglang/python/sglang/srt/managers/scheduler_output_processor_mixin.py", line 200, in process_batch_result_decode
    self.tp_worker.resolve_last_batch_result(launch_done)
  File "/usr/local/src/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 193, in resolve_last_batch_result
    copy_done.synchronize()
  File "/usr/local/lib/python3.10/site-packages/torch/cuda/streams.py", line 224, in synchronize
    super().synchronize()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


[2025-05-25 19:45:03] [ERROR] TpModelWorkerClient hit an exception: Traceback (most recent call last):
  File "/usr/local/src/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 118, in forward_thread_func
    self.forward_thread_func_()
  File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/src/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 151, in forward_thread_func_
    self.worker.forward_batch_generation(
  File "/usr/local/src/sglang/python/sglang/srt/managers/tp_worker.py", line 202, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
  File "/usr/local/src/sglang/python/sglang/srt/model_executor/model_runner.py", line 1185, in forward
    output = self._forward_raw(
  File "/usr/local/src/sglang/python/sglang/srt/model_executor/model_runner.py", line 1212, in _forward_raw
    ret = self.forward_decode(forward_batch, pp_proxy_tensors=pp_proxy_tensors)
  File "/usr/local/src/sglang/python/sglang/srt/model_executor/model_runner.py", line 1133, in forward_decode
    return self.model.forward(
  File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 1704, in forward
    hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 1608, in forward
    hidden_states, residual = layer(
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 1449, in forward
    hidden_states = self.self_attn(
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 795, in forward
    return self.forward_core(s)
  File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 844, in forward_core
    return self.forward_absorb_core(*inner_state)
  File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 1028, in forward_absorb_core
    output, _ = self.o_proj(attn_output)
                                                            
  File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 1449, in forward
    hidden_states = self.self_attn(
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 795, in forward
    return self.forward_core(s)
  File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 844, in forward_core
    return self.forward_absorb_core(*inner_state)
  File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 1028, in forward_absorb_core
    output, _ = self.o_proj(attn_output)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/src/sglang/python/sglang/srt/layers/linear.py", line 1291, in forward
    output_parallel = self.quant_method.apply(self, input_parallel, bias=bias_)
  File "/usr/local/src/sglang/python/sglang/srt/layers/quantization/fp8.py", line 420, in apply
    return apply_w8a8_block_fp8_linear(
  File "/usr/local/src/sglang/python/sglang/srt/layers/quantization/fp8_utils.py", line 173, in apply_w8a8_block_fp8_linear
    output = w8a8_block_fp8_matmul(
  File "/usr/local/src/sglang/python/sglang/srt/layers/quantization/fp8_kernel.py", line 786, in w8a8_block_fp8_matmul
    torch.ops.sglang.deep_gemm_fp8_fp8_bf16_nt(A, As, B, Bs, C)
  File "/usr/local/lib/python3.10/site-packages/torch/_ops.py", line 1123, in __call__
    return self._op(*args, **(kwargs or {}))
  File "/usr/local/src/sglang/python/sglang/srt/layers/quantization/fp8_kernel.py", line 80, in deep_gemm_fp8_fp8_bf16_nt
    deep_gemm_gemm_nt_f8f8bf16((A, As), (B, Bs), C)
  File "/usr/local/src/sglang/python/sglang/srt/layers/quantization/deep_gemm.py", line 368, in gemm_nt_f8f8bf16
    deep_gemm.gemm_fp8_fp8_bf16_nt(lhs, rhs, out)
  File "/usr/local/lib/python3.10/site-packages/deep_gemm/jit_kernels/gemm.py", line 196, in gemm_fp8_fp8_bf16_nt
    tensor_map_a = make_2d_tma_a_desc(
  File "/usr/local/lib/python3.10/site-packages/deep_gemm/jit_kernels/runtime.py", line 107, in make_2d_tma_a_desc
    return make_2d_tma_desc(global_address, Layout.RowMajor,
  File "/usr/local/lib/python3.10/site-packages/deep_gemm/jit_kernels/runtime.py", line 96, in make_2d_tma_desc
    return make_2d_tma_copy_desc(global_address, gmem_dim, cbd.cuuint64_t(gmem_cols * global_address.element_size()), smem_dim, swizzle_type)
  File "/usr/local/lib/python3.10/site-packages/deep_gemm/jit_kernels/runtime.py", line 85, in make_2d_tma_copy_desc
    raise Exception(f'Failed to encode tensor map: {res}')
Exception: Failed to encode tensor map: 700

During handling of the above exception, another exception occurred:

Reproduction

main branch pd
deepseekv3 0324 prefill tp16 dp4 decode tp32 dp32

Environment

main branch pd
deepseekv3 0324

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

@fzyzcjy hi, I found my error is related with --enable-dp-lm-head, and I add server args check in https://github.com/sgl-project/sglang/pull/6656,

it is not easy to repro…