tensorflow: Can't restore previous models

I installed Tensoflow from source. I can’t restore previous models saved with precompiled Tensorflow (installed via pip). I get lots of these:

Caused by op u'save/RestoreV2_56', defined at:
  File "grid-blstm.py", line 144, in <module>
    loadModel(session, './'+saveDir+'/')
  File "/home/user/Testing/DeepLearning/debug/grid-blstm/utils.py", line 267, in loadModel
    saver = tf.train.Saver()
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1067, in __init__
    self.build()
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1097, in build
    restore_sequentially=self._restore_sequentially)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 691, in build
    restore_sequentially, reshape)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 669, in restore_v2
    dtypes=dtypes, name=name)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2402, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1264, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/GridRNNCell/recurrent/cell_0/lstm_cell/w_i_diag not found in checkpoint. Found: 
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/project_c_0 (DT_DOUBLE) [20,100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/project_c_0/Adam (DT_DOUBLE) [20,100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/project_c_0/Adam_1 (DT_DOUBLE) [20,100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/project_m_0 (DT_DOUBLE) [20,100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/project_m_0/Adam (DT_DOUBLE) [20,100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/project_m_0/Adam_1 (DT_DOUBLE) [20,100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/B (DT_DOUBLE) [400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/B/Adam (DT_DOUBLE) [400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/B/Adam_1 (DT_DOUBLE) [400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_0 (DT_DOUBLE) [200,400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_0/Adam (DT_DOUBLE) [200,400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_0/Adam_1 (DT_DOUBLE) [200,400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_F_diag (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_F_diag/Adam (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_F_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_I_diag (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_I_diag/Adam (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_I_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_O_diag (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_O_diag/Adam (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_O_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/B (DT_DOUBLE) [400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/B/Adam (DT_DOUBLE) [400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/B/Adam_1 (DT_DOUBLE) [400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_0 (DT_DOUBLE) [200,400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_0/Adam (DT_DOUBLE) [200,400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_0/Adam_1 (DT_DOUBLE) [200,400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_F_diag (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_F_diag/Adam (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_F_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_I_diag (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_I_diag/Adam (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_I_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_O_diag (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_O_diag/Adam (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_O_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/project_c_0 (DT_DOUBLE) [20,100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/project_c_0/Adam (DT_DOUBLE) [20,100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/project_c_0/Adam_1 (DT_DOUBLE) [20,100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/project_m_0 (DT_DOUBLE) [20,100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/project_m_0/Adam (DT_DOUBLE) [20,100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/project_m_0/Adam_1 (DT_DOUBLE) [20,100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/B (DT_DOUBLE) [400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/B/Adam (DT_DOUBLE) [400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/B/Adam_1 (DT_DOUBLE) [400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_0 (DT_DOUBLE) [200,400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_0/Adam (DT_DOUBLE) [200,400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_0/Adam_1 (DT_DOUBLE) [200,400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_F_diag (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_F_diag/Adam (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_F_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_I_diag (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_I_diag/Adam (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_I_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_O_diag (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_O_diag/Adam (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_O_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/B (DT_DOUBLE) [400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/B/Adam (DT_DOUBLE) [400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/B/Adam_1 (DT_DOUBLE) [400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_0 (DT_DOUBLE) [200,400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_0/Adam (DT_DOUBLE) [200,400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_0/Adam_1 (DT_DOUBLE) [200,400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_F_diag (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_F_diag/Adam (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_F_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_I_diag (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_I_diag/Adam (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_I_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_O_diag (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_O_diag/Adam (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_O_diag/Adam_1 (DT_DOUBLE) [100]
Variable (DT_DOUBLE) [100,90]
Variable/Adam (DT_DOUBLE) [100,90]
Variable/Adam_1 (DT_DOUBLE) [100,90]
Variable_1 (DT_DOUBLE) [100,90]
Variable_1/Adam (DT_DOUBLE) [100,90]
Variable_1/Adam_1 (DT_DOUBLE) [100,90]
beta1_power (DT_FLOAT) []
beta2_power (DT_FLOAT) []

	 [[Node: save/RestoreV2_56 = RestoreV2[dtypes=[DT_DOUBLE], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_56/tensor_names, save/RestoreV2_56/shape_and_slices)]]
	 [[Node: save/RestoreV2_66/_91 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_172_save/RestoreV2_66", tensor_type=DT_DOUBLE, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Any idea?

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 25 (15 by maintainers)

Most upvoted comments

Hi, sorry for the delay. I thought I’d have time to look at this but apparently I do not! @martinwicke do you know who can look this? (This = a checkpoint from 1.0 can’t be loaded from 1.3, for more details start reading here: https://github.com/tensorflow/tensorflow/issues/7490#issuecomment-332373089)