onnxruntime: Wrong requested shape after a few thousand inference steps when using CUDA
Describe the bug
After many (typically many thousand) successful inference steps with the same data, the ONNX Runtime with CUDA suddenly stops with an error. The following error message suggests that a value set by an initializer has changed its value which yields an invalid requested shape in a Reshape node:
[E:onnxruntime:, sequential_executor.cc:309 Execute] Non-zero status code returned while running Reshape node. Name:'Reshape_15' Status Message: /code/onnxruntime/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:43 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, std::vector<long int>&) gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{4}, requested shape:{4,2,2}
This problem does not occur with the CPU or TensorRT version of ONNX Runtime.
This is what the graph of the ONNX model looks like (I suspect %19 is the issue here):
graph torch-jit-export (
%shape[INT64, 2]
) initializers (
%19[INT64, 1]
) {
%1 = Constant[value = <Scalar Tensor []>]()
%2 = Gather[axis = 0](%shape, %1)
%3 = Constant[value = <Scalar Tensor []>]()
%4 = Gather[axis = 0](%shape, %3)
%5 = Mul(%2, %4)
%6 = Unsqueeze[axes = [0]](%5)
%7 = Concat[axis = 0](%6)
%8 = ConstantOfShape[value = <Tensor>](%7)
%9 = Constant[value = <Scalar Tensor []>]()
%10 = Gather[axis = 0](%shape, %9)
%11 = Constant[value = <Scalar Tensor []>]()
%12 = Gather[axis = 0](%shape, %11)
%15 = Unsqueeze[axes = [0]](%10)
%16 = Unsqueeze[axes = [0]](%12)
%17 = Concat[axis = 0](%19, %15, %16)
%output = Reshape(%8, %17)
return %output
}
The input is always a one-dimensional array with value [2,2].
The ONNX model was created from the following PyTorch code (using PyTorch 1.6.0):
def forward(self, shape):
r = torch.zeros(shape[0] * shape[1])
return r.view(1, shape[0], shape[1])
System information
- OS Platform and Distribution: Container created from Dockerfile.cuda (problem also occurs in containers based on Ubuntu 18.04):
- ONNX Runtime installed from (source or binary): from source (GitHub master branch)
- ONNX Runtime version: 1.4.0
- Python version: 3.7.0
- GCC/Compiler version: 7.4.0
- CUDA/cuDNN version: CUDA 10.1 / cuDNN 7
- GPU model and memory: Nvidia GeForce GTX 1060 (problem also occurs on 1080Ti with 11GB)
To Reproduce
Run the following code in an environment (or container) with the ONNX Runtime with CUDA (not TensorRT) from a directory that contains the file issue.onnx that is attached to this issue:
#!/usr/bin/env python3
"""Minimal example to reproduce issue with with ONNX Runtime on GPU"""
import numpy as np
import onnxruntime
def create_inputs():
"""create input for the model as a numpy array"""
return np.array([2, 2])
def run_onnx(file_name):
"""run ONNX model until it fails (when run on GPU)
on my computer this tends to fail withing the first 100000 iterations
"""
options = onnxruntime.SessionOptions()
session = onnxruntime.InferenceSession(file_name, options)
shape = create_inputs()
for iteration in range(int(1e6)):
try:
result = session.run(output_names=['output'],
input_feed={
'shape': shape,
})
except Exception as e:
print(f"\nerror occured during iteration {iteration}")
break
return result
def main():
"""try to run inference"""
filename = 'issue.onnx'
result = run_onnx(filename)
print(f"result: {result[0].shape}")
if __name__ == '__main__':
main()
After a few seconds and a few thousand (sometimes tens of thousands of) iterations the loop in run_onnx() should abort with the error message above. While I have managed to reproduce this issue on multiple systems in various configurations, the number of iterations before the error occurs has varied dramatically. Any sort of additional GPU load (from watching Youtube videos to multiplying large matrices) seems to make the issue more reproducible.
Expected behavior Inference should work deterministically.
Additional context The ONNX/PyTorch model provided here may not be very useful in itself. However, the same issue also appears in more complex models that contain similar steps. Larger models seem to require fewer iterations until the error occurs.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 6
- Comments: 16 (7 by maintainers)
Commits related to this issue
- Add CUDA option to run copy in default stream This change fixes #4829. Thanks @maherzog for providing the repro! The bug is caused by memory reuse in BFC arena, where copy and compute stream in CUDA... — committed to microsoft/onnxruntime by deleted user 4 years ago
- Add CUDA option to run copy in default stream (#5445) * Add CUDA option to run copy in default stream This change fixes #4829. Thanks @maherzog for providing the repro! The bug is caused by mem... — committed to microsoft/onnxruntime by deleted user 4 years ago
Sorry for being later in the discussion, and thanks @maherzog for this repro. As @HectorSVC pointed out, the bug is indeed caused by memory reuse. The issue is that copy and compute stream in CUDA has a racing condition in BFC arena. BFC arena is an arena allocator on top of cudaMalloc/Free to reduce the cost in syncing CPU and GPU when alloc/free.
To make CPU and GPU running asynchronously, buffers freed on CPU could still be in use on GPU. This is OK if there’s only one stream, where the execution order in CPU and GPU are consistent. For example, if we have two kernels A and B, when CPU runs with order of allocA->computeA->freeA->allocB->computeB->freeB, even when A and B shares the same memory, computeA and computeB will not have racing in the same GPU compute stream. However, if it is allocA->CopyA->freeA->allocB->computeB->freeB in CPU, the order of execution in GPU could have copyA happen after computeB, when copy and compute happens in different GPU streams.
For this particular case, the execution plan in CPU is:
Here in step [5], the input buffer to MemcpyToHost is freed to BFC arena, and then allocated by static allocation plan to the output of step [8]. Because compute stream and copy stream runs concurrently on GPU, the GPU execution order may not match CPU’s plan, and thus causing memory to be overwritten.
In this repro, if we add a line of code to disable memory pattern, the problem is gone:
However, disabling memory pattern is not a full solution to the racing of streams in BFC arena. As a short term fix, we might force the copy stream to be the same as compute stream, by changing here to:
And also remove the cudaStreamDestory in the dtor. This approach might cause some performance degradation for certain models though. A thorough fix to BFC arena to support multiple stream is being looked at, and once that is in, we can continue to have the concurrent copy and compute streams.
There’s nothing wrong with CUDA Concat implementation, should be relate to memory re-use. Still debugging.
Thank you. I can repro the bug.