deepspeech.pytorch: Segmentation fault during training (Volta, others)

For the first epoch, the batches are samples in increasing sequence length order, so you progressively need more and more memory. What happens is before the failing cudnn call there’s very little memory left on the GPU (in my tests, 300 K, you can confirm by adding

lib = ctypes.cdll.LoadLibrary(None)
free_mem = ctypes.c_long()
total_mem = ctypes.c_long()
lib.cudaMemGetInfo(ctypes.byref(free_mem), ctypes.byref(total_mem))
print("free mem", free_mem.value)

before cudnnRNNForwardTraining call. Cuda requires some free memory for internal operation (stack space for kernels, device memory for events etc) and when so little memory is left something is not handled correctly. (FWIW, same thing can happen on the CPU when you are pushing the memory limits, allocation can be reported as succeeded, and when you try to use it, you’d hang or segfault). My output for the code modified as above, run on a single GPU with the batch size of 64 is

Epoch: [1][515/1422]	Time 1.177 (0.583)	Data 0.008 (0.006)	Loss 172.8604 (119.4332)	
free mem 7252279296
reserve size 801177600 1105817763840 workspace_size 313344000 1107793281024 input size torch.Size([326, 64, 672])
free mem 7252279296
reserve size 801177600 1106620973056 workspace_size 315801600 1107793281024 input size torch.Size([326, 64, 800])
free mem 7252279296
reserve size 801177600 1108502118400 workspace_size 315801600 1107793281024 input size torch.Size([326, 64, 800])
free mem 7252279296
reserve size 801177600 1117253533696 workspace_size 315801600 1107793281024 input size torch.Size([326, 64, 800])
free mem 7252279296
reserve size 801177600 1118056742912 workspace_size 315801600 1107926843392 input size torch.Size([326, 64, 800])
Epoch: [1][516/1422]	Time 0.915 (0.584)	Data 0.008 (0.006)	Loss 151.3297 (119.4950)	
free mem 6446972928
reserve size 803635200 1109305327616 workspace_size 314163200 1107793281024 input size torch.Size([327, 64, 672])
free mem 5641666560
reserve size 803635200 1110110633984 workspace_size 316620800 1107793281024 input size torch.Size([327, 64, 800])
free mem 4836360192
reserve size 803635200 1110915940352 workspace_size 316620800 1107793281024 input size torch.Size([327, 64, 800])
free mem 4031053824
reserve size 803635200 1111721246720 workspace_size 316620800 1107793281024 input size torch.Size([327, 64, 800])
free mem 3225747456
reserve size 803635200 1112526553088 workspace_size 316620800 1107927236608 input size torch.Size([327, 64, 800])
Epoch: [1][517/1422]	Time 0.930 (0.585)	Data 0.007 (0.006)	Loss 165.3099 (119.5836)	
free mem 2420441088
reserve size 803635200 1109305327616 workspace_size 314163200 1107793281024 input size torch.Size([327, 64, 672])
free mem 2420441088
reserve size 803635200 1110110633984 workspace_size 316620800 1107793281024 input size torch.Size([327, 64, 800])
free mem 2420441088
reserve size 803635200 1110915940352 workspace_size 316620800 1107793281024 input size torch.Size([327, 64, 800])
free mem 2420441088
reserve size 803635200 1111721246720 workspace_size 316620800 1107793281024 input size torch.Size([327, 64, 800])
free mem 2420441088
reserve size 803635200 1112526553088 workspace_size 316620800 1107927236608 input size torch.Size([327, 64, 800])
Epoch: [1][518/1422]	Time 0.919 (0.585)	Data 0.008 (0.006)	Loss 179.5324 (119.6994)	
free mem 1613037568
reserve size 806092800 1114137165824 workspace_size 314982400 1107793281024 input size torch.Size([328, 64, 672])
free mem 805634048
reserve size 806092800 1114944569344 workspace_size 317440000 1107793281024 input size torch.Size([328, 64, 800])
free mem 327680
reserve size 806092800 1115751972864 workspace_size 317440000 1107793281024 input size torch.Size([328, 64, 800])
Segmentation fault (core dumped)

Ignore “reserve size” lines, I was checking that allocations for reserve and workspace look ok. So, there are a few things to blame for this outcome

1. CUDA should handle oom more gracefully than just segfaulting.
2. IIRC, there are still some unresolved issues with pytorch using more memory for RNNs than it should
3. Monotonically increasing sequences, as in this example, are particularly bad for caching allocator, it cannot reuse blocks and is forced to constantly free and reallocate them.
4. Tedlium dataset has pretty long sequences, I segfaulted at less than 40% of the epoch, and sequences were of length 328 already.

Emptying memory allocator cache was exposed recently in the upstream pytorch, may be doing that, or collecting garbage before each iteration will allow you to move a bit further. Or using smaller batch, or using smaller hidden state of LSTM, in short, all the standard things that you would do to try to fit your problem in memory.