sglang: [Bug] main pd version Exception: Failed to encode tensor map: 700
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
[2025-05-25 19:45:03] [ERROR] Scheduler hit an exception: Traceback (most recent call last):
File "/usr/local/src/sglang/python/sglang/srt/managers/scheduler.py", line 2346, in run_scheduler_process
scheduler.event_loop_overlap_disagg_decode()
File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/usr/local/src/sglang/python/sglang/srt/disaggregation/decode.py", line 558, in event_loop_overlap_disagg_decode
self.process_batch_result(tmp_batch, tmp_result)
File "/usr/local/src/sglang/python/sglang/srt/managers/scheduler.py", line 1626, in process_batch_result
self.process_batch_result_decode(batch, result, launch_done)
File "/usr/local/src/sglang/python/sglang/srt/managers/scheduler_output_processor_mixin.py", line 200, in process_batch_result_decode
self.tp_worker.resolve_last_batch_result(launch_done)
File "/usr/local/src/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 193, in resolve_last_batch_result
copy_done.synchronize()
File "/usr/local/lib/python3.10/site-packages/torch/cuda/streams.py", line 224, in synchronize
super().synchronize()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
[2025-05-25 19:45:03] [ERROR] TpModelWorkerClient hit an exception: Traceback (most recent call last):
File "/usr/local/src/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 118, in forward_thread_func
self.forward_thread_func_()
File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/usr/local/src/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 151, in forward_thread_func_
self.worker.forward_batch_generation(
File "/usr/local/src/sglang/python/sglang/srt/managers/tp_worker.py", line 202, in forward_batch_generation
logits_output, can_run_cuda_graph = self.model_runner.forward(
File "/usr/local/src/sglang/python/sglang/srt/model_executor/model_runner.py", line 1185, in forward
output = self._forward_raw(
File "/usr/local/src/sglang/python/sglang/srt/model_executor/model_runner.py", line 1212, in _forward_raw
ret = self.forward_decode(forward_batch, pp_proxy_tensors=pp_proxy_tensors)
File "/usr/local/src/sglang/python/sglang/srt/model_executor/model_runner.py", line 1133, in forward_decode
return self.model.forward(
File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 1704, in forward
hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 1608, in forward
hidden_states, residual = layer(
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 1449, in forward
hidden_states = self.self_attn(
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 795, in forward
return self.forward_core(s)
File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 844, in forward_core
return self.forward_absorb_core(*inner_state)
File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 1028, in forward_absorb_core
output, _ = self.o_proj(attn_output)
File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 1449, in forward
hidden_states = self.self_attn(
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 795, in forward
return self.forward_core(s)
File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 844, in forward_core
return self.forward_absorb_core(*inner_state)
File "/usr/local/src/sglang/python/sglang/srt/models/deepseek_v2.py", line 1028, in forward_absorb_core
output, _ = self.o_proj(attn_output)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/src/sglang/python/sglang/srt/layers/linear.py", line 1291, in forward
output_parallel = self.quant_method.apply(self, input_parallel, bias=bias_)
File "/usr/local/src/sglang/python/sglang/srt/layers/quantization/fp8.py", line 420, in apply
return apply_w8a8_block_fp8_linear(
File "/usr/local/src/sglang/python/sglang/srt/layers/quantization/fp8_utils.py", line 173, in apply_w8a8_block_fp8_linear
output = w8a8_block_fp8_matmul(
File "/usr/local/src/sglang/python/sglang/srt/layers/quantization/fp8_kernel.py", line 786, in w8a8_block_fp8_matmul
torch.ops.sglang.deep_gemm_fp8_fp8_bf16_nt(A, As, B, Bs, C)
File "/usr/local/lib/python3.10/site-packages/torch/_ops.py", line 1123, in __call__
return self._op(*args, **(kwargs or {}))
File "/usr/local/src/sglang/python/sglang/srt/layers/quantization/fp8_kernel.py", line 80, in deep_gemm_fp8_fp8_bf16_nt
deep_gemm_gemm_nt_f8f8bf16((A, As), (B, Bs), C)
File "/usr/local/src/sglang/python/sglang/srt/layers/quantization/deep_gemm.py", line 368, in gemm_nt_f8f8bf16
deep_gemm.gemm_fp8_fp8_bf16_nt(lhs, rhs, out)
File "/usr/local/lib/python3.10/site-packages/deep_gemm/jit_kernels/gemm.py", line 196, in gemm_fp8_fp8_bf16_nt
tensor_map_a = make_2d_tma_a_desc(
File "/usr/local/lib/python3.10/site-packages/deep_gemm/jit_kernels/runtime.py", line 107, in make_2d_tma_a_desc
return make_2d_tma_desc(global_address, Layout.RowMajor,
File "/usr/local/lib/python3.10/site-packages/deep_gemm/jit_kernels/runtime.py", line 96, in make_2d_tma_desc
return make_2d_tma_copy_desc(global_address, gmem_dim, cbd.cuuint64_t(gmem_cols * global_address.element_size()), smem_dim, swizzle_type)
File "/usr/local/lib/python3.10/site-packages/deep_gemm/jit_kernels/runtime.py", line 85, in make_2d_tma_copy_desc
raise Exception(f'Failed to encode tensor map: {res}')
Exception: Failed to encode tensor map: 700
During handling of the above exception, another exception occurred:
Reproduction
main branch
pd
deepseekv3 0324
prefill tp16 dp4
decode tp32 dp32
Environment
main branch
pd
deepseekv3 0324
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 15 (7 by maintainers)
@fzyzcjy hi, I found my error is related with --enable-dp-lm-head, and I add server args check in https://github.com/sgl-project/sglang/pull/6656,
it is not easy to repro…