oneDNN: How to debug conv2d backward `Unexpected page fault` error
Hi, I recently encountered an issue related to conv2d backward.
After updating oneDNN to 3.7.0, conv2d backward throws Unexpected page fault after a specific shape matmul operation.
I tried to reproduce this issue with C++ according oneDNN verbose logs from run-able tensor shape, but couldn’t replicate the issue.
I guess it’s caused by operator inconsistencies generated under specific shapes.
Could I get your suggestions on how to further investigate or debug this issue?
Pytorch reproducer
import os
import torch
import torch.nn as nn
from torch.testing._internal.common_utils import TestCase
cpu_device = torch.device("cpu")
dpcpp_device = torch.device("xpu")
a_size = int(os.environ.get('A_SIZE', 1))
# This reproducer will throw page fault when a_size=1~8
a=torch.rand([a_size, 1024], device='xpu:0', dtype=torch.float16)
b=torch.rand([1024, 1024], device='xpu:0', dtype=torch.float16)
torch.mm(a,b)
dtype=torch.double
x_cpu = torch.randn(
[1, 64, 256, 256], dtype=dtype, device=cpu_device, requires_grad=True
)
grad_cpu = torch.full(
[1, 64, 256, 256], 1e-3, dtype=dtype, device=cpu_device, requires_grad=True
)
conv_cpu = nn.Conv2d(
64, 64, kernel_size=3, stride=1, padding=1, bias=False
).double()
x_dpcpp = x_cpu.to(dpcpp_device).requires_grad_()
grad_dpcpp = grad_cpu.to(dpcpp_device)
conv_dpcpp = conv_cpu.to(dpcpp_device)
y_dpcpp = conv_dpcpp(x_dpcpp)
y_dpcpp.backward(grad_dpcpp)
oneDNN Verbose
onednn_verbose,v1,info,oneDNN v3.7.0 (commit 5e9224036021433d2577548ed0539fe9a53256bc)
onednn_verbose,v1,info,cpu,runtime:threadpool,nthr:56
onednn_verbose,v1,info,cpu,isa:Intel AVX-512 with float16, Intel DL Boost and bfloat16 support
onednn_verbose,v1,info,gpu,runtime:DPC++
onednn_verbose,v1,info,gpu,engine,sycl gpu device count:8
onednn_verbose,v1,info,gpu,engine,0,backend:Level Zero,name:Intel(R) Data Center GPU Max 1550,driver_version:1.6.31294,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,1,backend:Level Zero,name:Intel(R) Data Center GPU Max 1550,driver_version:1.6.31294,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,2,backend:Level Zero,name:Intel(R) Data Center GPU Max 1550,driver_version:1.6.31294,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,3,backend:Level Zero,name:Intel(R) Data Center GPU Max 1550,driver_version:1.6.31294,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,4,backend:Level Zero,name:Intel(R) Data Center GPU Max 1550,driver_version:1.6.31294,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,5,backend:Level Zero,name:Intel(R) Data Center GPU Max 1550,driver_version:1.6.31294,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,6,backend:Level Zero,name:Intel(R) Data Center GPU Max 1550,driver_version:1.6.31294,binary_kernels:enabled
onednn_verbose,v1,info,gpu,engine,7,backend:Level Zero,name:Intel(R) Data Center GPU Max 1550,driver_version:1.6.31294,binary_kernels:enabled
onednn_verbose,v1,info,graph,backend,0:dnnl_backend
onednn_verbose,v1,primitive,info,template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc
onednn_verbose,v1,graph,info,template:operation,engine,partition_id,partition_kind,op_names,data_formats,logical_tensors,fpmath_mode,implementation,backend
onednn_verbose,v1,primitive,exec,gpu:0,matmul,jit:gemm:any,undef,src:f16::blocked:ab::f0 wei:f16::blocked:ab::f0 dst:f16::blocked:ab::f0,attr-scratchpad:user,,1x1024:1024x1024
onednn_verbose,v1,primitive,exec,gpu:0,convolution,jit:ir,forward_training,src:f64::blocked:abcd::f0 wei:f64::blocked:abcd::f0 bia:undef::undef::: dst:f64::blocked:abcd::f0,attr-scratchpad:user,alg:convolution_direct,mb1_ic64oc64_ih256oh256kh3sh1dh0ph1_iw256ow256kw3sw1dw0pw1
FATAL: Unexpected page fault from GPU at 0xff0000000e97a000, ctx_id: 1 (CCS) type: 0 (NotPresent), level: 1 (PDE), access: 0 (Read), banned: 1, aborting.
FATAL: Unexpected page fault from GPU at 0xff0000000e97a000, ctx_id: 1 (CCS) type: 0 (NotPresent), level: 1 (PDE), access: 0 (Read), banned: 1, aborting.
Abort was called at 288 line in file:
./shared/source/os_interface/linux/drm_neo.cpp
Aborted
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (12 by maintainers)
@hoshibara @rcao8 Thanks. I was able to reproduce it and reduce to two primitives: MatMul (called once) and deconvolution (in a loop). The loop part has only one kernel (deconvolution) and doesn’t have any kernel creations or memory allocations beyond the first iteration. So we basically execute the same kernel with the same buffers over and over again. The fact that it occasionally generates a page fault after some time points to a hardware bug. I’ll work on a C++ reproducer next week to try different things to understand what triggers this behavior.
Page fault:
@hoshibara @rcao8 We were able to confirm that the PVC HW bug @echeresh mentioned is the culprit. The linked PR (#2935) should fix the issue you’re seeing.
@hoshibara @rcao8 The preliminary conclusion is that this is caused by a known double data type related hardware bug in PVC. I was able to reproduce it in benchdnn so it’s completely on our side from this point - we’ll investigate it and prepare a fix. A fix should be available in some future oneDNN release.
As a mitigation, I see that overriding the thread arbitration policy makes the issue non-reproducible but it may come with a performance cost and may still produce the same issue but at a more rare rate:
You’d need two convolutions at least, on forward and one backward data (and maybe one backward weights as well), according to the messages above.
@hoshibara Thanks, the faulting address looks similar, ~35 MB after the scratchpad bounds.
Okay, I guess PyTorch doesn’t support OpenCL backend…
Yes - disable
gen_convkernel orconv_reorderkernels. For that you would need to adjust oneDNN sources. If PyTorch links with oneDNN dynamically (libdnnl.sodependency) - I think we can easily debug it if it reproduces on our side.So far, I have one suggestion only: can you please collect and share oneDNN kernel dumps (
*.binfiles) - for that you need to setONEDNN_JIT_DUMP=1. I can compare binaries with the binaries on my side to see if kernels are different (they should be exactly the same).If this doesn’t clarify anything, as the next thing, I would try to reproduce and debug it on our side - for that can you please share links/steps you used to install and run PyTorch?
Can’t reproduce it on MTL iGPU for oneDNN v3.7.0, but reproduced on Intel® Data Center GPU Max 1100,driver_version:1.6.33022 with oneDNN v3.7.1 (commit https://github.com/uxlfoundation/oneDNN/commit/8d263e693366ef8db40acc569cc7d8edf644556d)
onednn_verbose,v1,info,oneDNN v3.7.1 (commit 8d263e693366ef8db40acc569cc7d8edf644556d) onednn_verbose,v1,info,cpu,runtime:threadpool,nthr:32 onednn_verbose,v1,info,cpu,isa:Intel AVX-512 with float16, Intel DL Boost and bfloat16 support and Intel AMX with bfloat16 and 8-bit integer support onednn_verbose,v1,info,gpu,runtime:DPC++ onednn_verbose,v1,info,gpu,engine,sycl gpu device count:2 onednn_verbose,v1,info,gpu,engine,0,backend:Level Zero,name:Intel(R) Data Center GPU Max 1100,driver_version:1.6.33022,binary_kernels:enabled onednn_verbose,v1,info,gpu,engine,1,backend:Level Zero,name:Intel(R) Data Center GPU Max 1100,driver_version:1.6.33022,binary_kernels:enabled onednn_verbose,v1,info,graph,backend,0:dnnl_backend onednn_verbose,v1,primitive,info,template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc,exec_time onednn_verbose,v1,graph,info,template:operation,engine,partition_id,partition_kind,op_names,data_formats,logical_tensors,fpmath_mode,implementation,backend,exec_time onednn_verbose,v1,primitive,exec,gpu:0,matmul,jit:gemm:any,undef,src:f16::blocked:ab::f0 wei:f16::blocked:ab::f0 dst:f16::blocked:ab::f0,attr-scratchpad:user,,1x1024:1024x1024,0.73291 onednn_verbose,v1,primitive,exec,gpu:0,convolution,jit:ir,forward_training,src:f64::blocked:abcd::f0 wei:f64::blocked:abcd::f0 bia:undef::undef::: dst:f64::blocked:abcd::f0,attr-scratchpad:user,alg:convolution_direct,mb1_ic64oc64_ih256oh256kh3sh1dh0ph1_iw256ow256kw3sw1dw0pw1,0.8479 Segmentation fault from GPU at 0xff0000000e6f3000, ctx_id: 1 (CCS) type: 0 (NotPresent), level: 1 (PDE), access: 0 (Read), banned: 1, aborting. Segmentation fault from GPU at 0xff0000000e6f3000, ctx_id: 1 (CCS) type: 0 (NotPresent), level: 1 (PDE), access: 0 (Read), banned: 1, aborting. Abort was called at 276 line in file: /opt/src/compute-neo/shared/source/os_interface/linux/drm_neo.cpp Aborted (core dumped)Hi @dzarukin, thank you for your patience and suggestions. I followed your advice and did further testing. Here are the results:
It was also mentioned it started after upgrade to v3.7. Does it imply the previous version was working well? And if so, which version was that?
I rollback the oneDNN integrated in the latest PyTorch and confirmed that this issue first appeared in
v3.7(5e9224036021433d2577548ed0539fe9a53256bc). When I switched to thev3.6.2(2eb3dd1082db767fab171e934c551c609008289a) branch, this test can be ran properly. In addition, I collectedONEDNN_VERBOSE=alllogs for both versions when running the reproducer. Follows are the differences:Does the issue happen if and only if there’s this specific matmul in there? What happens if matmul op gets removed?
Yes. If I remove the
torch.mmoperation in the code, conv2d runs without any errors.Does the issue happen with f16 matmul on any other shape? Like 1x256:256x256? Or maybe 16x16:16x16?
Yes. The smallest shape I found that can reproduce the issue is
(1~8)x8 : 8x4.Does it happen with f32 or f64 matmul used instead of f16?
No. If I use f32 or f64 for matmul with the same tensor shapes, conv2d works fine and no error occurs.
Does it happen for this specific convolution shape? Will any other fail?
Yes. The smallest convolution shape I found so far that reproduces the issue is as follows:
Thank you again for your time and support!