tensorflow: keras LSTM Fail to find the dnn implementation

System information

CUDA/cuDNN version: 10.1
GPU model and memory: GeForce RTX 2080
TF 2.1.0:

uncommenting the LSTM layer will yield the following error:

UnknownError:  [_Derived_]  Fail to find the dnn implementation.
	 [[{{node CudnnRNN}}]]
	 [[sequential_6/bidirectional_2/backward_lstm_3/StatefulPartitionedCall]]
	 [[Reshape_11/_38]] [Op:__inference_distributed_function_39046]

working code:

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(encoder.vocab_size, 64),
    #tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(loss='binary_crossentropy',
              optimizer=tf.keras.optimizers.Adam(1e-4),
              metrics=['accuracy'])
history = model.fit(train_dataset, epochs=10,
                    validation_data=test_dataset, 
                    validation_steps=30)

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 50 (7 by maintainers)

Most upvoted comments

@Lay4U @ARozental Please use the below code while importing tensorflow and let me know if the issue still persists. Thanks!

import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], enable=True)

+102

gowthamkpr on Feb 11, 2020

@gowthamkpr It doesn’t help

alonRozental on Feb 16, 2020

gpus = tf.config.experimental.list_physical_devices(device_type='GPU')
tf.config.experimental.set_visible_devices(devices=gpus[1], device_type='GPU')
tf.config.experimental.set_memory_growth(device=gpus[1], enable=True)

above work for me.

shaoeChen on Jun 3, 2020

Just a heads up I had this error but I noticed in the output this error as well

Loaded runtime CuDNN library: 7.1.3 but source was compiled with: 7.6.4.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version.

Resolved by updating my conda env with

conda install -c anaconda cudnn

ElliotVilhelm on May 14, 2020

@Lay4U @ARozental Please use the below code while importing tensorflow and let me know if the issue still persists. Thanks!
import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], enable=True)

this resolved the issue in my case… Using:

CUDA 12.2
tf.version is 2.1.0
tf.keras.version is: 2.2.4-tf
Python 3.7.4
cuDNN v7.6.5 (November 18th, 2019), for CUDA 10.2

pavlexander on Mar 29, 2020

Same issue here I tried all the aforementioned solutions. None seems to resolve the issue

trifwn on Apr 22, 2021

I got the same problem, which is solved by this. Thanks a lot!

@Lay4U @ARozental Please use the below code while importing tensorflow and let me know if the issue still persists. Thanks!
import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], enable=True)

zhongruiwang on May 11, 2020

I confirm that it does not help

olalakul on Mar 1, 2020

@Saduf2019 I’m running TF 2.1.0. I don’t think the problem exists in TF1 which is used in the notebook. also making the following change makes the code work:

    #tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
    tf.keras.layers.Bidirectional(tf.keras.layers.RNN(tf.keras.layers.LSTMCell(64))),

I would think that those 2 lines should do the same thing (please correct me if I’m wrong) but it seems only the second line works.

alonRozental on Feb 10, 2020

@Lay4U @ARozental Please use the below code while importing tensorflow and let me know if the issue still persists. Thanks!
import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], enable=True)

RuntimeError: Physical devices cannot be modified after being initialized

this-is-shashank on Jul 4, 2021

@Lay4U @ARozental Please use the below code while importing tensorflow and let me know if the issue still persists. Thanks!
import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], enable=True)

Just had the same issue here, managed to fix with this solution

My setup: Windows 10 CUDA 11.2 Tensorflow 2.3 Nvidia Driver 460.x Geforce RTX 2060 Python 3.8

marcelmotta on Feb 10, 2021

conda install -c anaconda cudnn

This worked for us when getting

tensorflow.python.framework.errors_impl.UnknownError:  [_Derived_]  Fail to find the dnn implementation.

Thanks @ElliotVilhelm

paulmwatson on Jul 28, 2020

Ok I managed to make it work after fighting with CUDA 10.1 and 10.2 (10.2 works nice with 2.3 nightly) for a while, environments, OS and everything.

Narrowed it to a seeming harmless line

I was running tf.test.gpu_device_name() to check there was a GPU and print its name. That command when run at any time makes the model fail on train with the mentioned error: Unknown: Fail to find the dnn implementation

The tf.config.experimental.set_visible_devices command that @shaoeChen mentioned didn’t change anything for me so I removed it.

I managed to make it work more reliably running this right after importing tensorflow (and other libs, but I don’t think it changes anything)

gpus = tf.config.experimental.list_physical_devices(device_type='GPU')
tf.config.experimental.set_memory_growth(device=gpus[0], enable=True)

Is this a known bug or some unintended behaviour?

sousandrei on Jun 6, 2020