tensorflow: Error occurs when importing metagraph that contains cudnnRNN cells

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS 7.4
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
• TensorFlow installed from (source or binary): pip install .whl
• TensorFlow version (use command below): 1.12.0
• Python version: 2.7.14
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source): c++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
• CUDA/cuDNN version: cuda: 9.0 cuDNN: 7.1.3
• GPU model and memory: GTX1050Ti/4GB tf_env.txt

Describe the current behavior I try to import graph from a MetaGraphDef proto that contains a CudnnRNN cell, the error below raises:

 Traceback (most recent call last):
  File "test.py", line 21, in <module>
    saver = tf.train.import_meta_graph(mgd)
  File "/root/anaconda2/envs/tf12/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1674, in import_meta_graph
    meta_graph_or_file, clear_devices, import_scope, **kwargs)[0]
  File "/root/anaconda2/envs/tf12/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1696, in _import_meta_graph_with_return_elements
    **kwargs))
  File "/root/anaconda2/envs/tf12/lib/python2.7/site-packages/tensorflow/python/framework/meta_graph.py", line 852, in import_scoped_meta_graph_with_return_elements
    ops.prepend_name_scope(value, scope_to_prepend_to_names))
  File "/root/anaconda2/envs/tf12/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3490, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/root/anaconda2/envs/tf12/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3550, in _as_graph_element_locked
    "graph." % repr(name))
KeyError: "The name 'cudnn_lstm/opaque_kernel_saveable' refers to an Operation not in the graph."

What I’ve done: I’ve debuged the souce code and found out that a CudnnLSTMSaveable instance is created when cudnnRNN cell gets called and corresponding op was saved in tf.GraphKeys.SAVEABLE_OBJECTS collections with the name “cudnn_lstm/opaque_kernel_saveable”. when executing saver.import_meta_graph, in saver.save_op(line 189 of saver.py) function the saveable object mentioned above is somehow transfered into “cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel” and “cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias” and gets saved. But the op named “cudnn_lstm/opaque_kernel_saveable” didn’t gets saved. I’m wandering if this is a bug.

Describe the expected behavior graph gets imported

Code to reproduce the issue

import tensorflow as tf
import numpy as np
num_layers = 1
num_units = 128
direction = "unidirectional"
inputs    = tf.placeholder(tf.float32, [None, None, 32], name="inputs")
convolved = tf.transpose(inputs, [1, 0, 2])

lstm      =  tf.contrib.cudnn_rnn.CudnnLSTM(num_layers, num_units, direction=direction)
outputs, output_states = lstm(convolved, training=True)
enc_output = tf.transpose(outputs, [1, 0, 2])
optimizer  = tf.train.AdamOptimizer(0.001)
optim = optimizer.minimize(enc_output)
saver     = tf.train.Saver()
init      = tf.global_variables_initializer()
with tf.Session() as session:
    session.run(init)
    # export MetaGraphDef proto to mgd
    mgd=saver.export_meta_graph()
with tf.Session(graph=tf.Graph()) as session:
    # import graph from mgd on another graph
    saver = tf.train.import_meta_graph(mgd)