deepspeech.pytorch: RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

this happened when i set cuda=True

)
Traceback (most recent call last):
  File "train.py", line 318, in <module>
    main()
  File "train.py", line 182, in main
    out = model(inputs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 61, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 71, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/parallel_apply.py", line 45, in parallel_apply
    raise output
RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 16 (4 by maintainers)

Most upvoted comments

Just sudo rm -rf ~/.nv and reboot, everything will be ok!

+41

softhuafei on Oct 27, 2017

@softhuafei It works without reboot.

lonelylingoes on Jan 22, 2018

thanks for you help! ubuntu16.04, cuda-8.0, install pytorch from source, success training with gpu if i set torch.backends.cudnn.enabled = False

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61

demobin8 on Apr 27, 2017

Thanks! Training with cudnn is about 10x faster.

demobin8 on Apr 28, 2017