tensorflow: Can't restore previous models
I installed Tensoflow from source. I can’t restore previous models saved with precompiled Tensorflow (installed via pip). I get lots of these:
Caused by op u'save/RestoreV2_56', defined at:
File "grid-blstm.py", line 144, in <module>
loadModel(session, './'+saveDir+'/')
File "/home/user/Testing/DeepLearning/debug/grid-blstm/utils.py", line 267, in loadModel
saver = tf.train.Saver()
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1067, in __init__
self.build()
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1097, in build
restore_sequentially=self._restore_sequentially)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 691, in build
restore_sequentially, reshape)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
[spec.tensor.dtype])[0])
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 669, in restore_v2
dtypes=dtypes, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2402, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1264, in __init__
self._traceback = _extract_stack()
NotFoundError (see above for traceback): Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/GridRNNCell/recurrent/cell_0/lstm_cell/w_i_diag not found in checkpoint. Found:
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/project_c_0 (DT_DOUBLE) [20,100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/project_c_0/Adam (DT_DOUBLE) [20,100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/project_c_0/Adam_1 (DT_DOUBLE) [20,100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/project_m_0 (DT_DOUBLE) [20,100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/project_m_0/Adam (DT_DOUBLE) [20,100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/project_m_0/Adam_1 (DT_DOUBLE) [20,100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/B (DT_DOUBLE) [400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/B/Adam (DT_DOUBLE) [400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/B/Adam_1 (DT_DOUBLE) [400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_0 (DT_DOUBLE) [200,400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_0/Adam (DT_DOUBLE) [200,400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_0/Adam_1 (DT_DOUBLE) [200,400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_F_diag (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_F_diag/Adam (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_F_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_I_diag (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_I_diag/Adam (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_I_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_O_diag (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_O_diag/Adam (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_O_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/B (DT_DOUBLE) [400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/B/Adam (DT_DOUBLE) [400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/B/Adam_1 (DT_DOUBLE) [400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_0 (DT_DOUBLE) [200,400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_0/Adam (DT_DOUBLE) [200,400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_0/Adam_1 (DT_DOUBLE) [200,400]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_F_diag (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_F_diag/Adam (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_F_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_I_diag (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_I_diag/Adam (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_I_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_O_diag (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_O_diag/Adam (DT_DOUBLE) [100]
BiRNN/BW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_O_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/project_c_0 (DT_DOUBLE) [20,100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/project_c_0/Adam (DT_DOUBLE) [20,100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/project_c_0/Adam_1 (DT_DOUBLE) [20,100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/project_m_0 (DT_DOUBLE) [20,100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/project_m_0/Adam (DT_DOUBLE) [20,100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/project_m_0/Adam_1 (DT_DOUBLE) [20,100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/B (DT_DOUBLE) [400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/B/Adam (DT_DOUBLE) [400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/B/Adam_1 (DT_DOUBLE) [400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_0 (DT_DOUBLE) [200,400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_0/Adam (DT_DOUBLE) [200,400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_0/Adam_1 (DT_DOUBLE) [200,400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_F_diag (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_F_diag/Adam (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_F_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_I_diag (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_I_diag/Adam (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_I_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_O_diag (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_O_diag/Adam (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_0/LSTMCell/W_O_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/B (DT_DOUBLE) [400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/B/Adam (DT_DOUBLE) [400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/B/Adam_1 (DT_DOUBLE) [400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_0 (DT_DOUBLE) [200,400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_0/Adam (DT_DOUBLE) [200,400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_0/Adam_1 (DT_DOUBLE) [200,400]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_F_diag (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_F_diag/Adam (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_F_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_I_diag (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_I_diag/Adam (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_I_diag/Adam_1 (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_O_diag (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_O_diag/Adam (DT_DOUBLE) [100]
BiRNN/FW/MultiRNNCell/Cell0/GridRNNCell/recurrent/cell_1/LSTMCell/W_O_diag/Adam_1 (DT_DOUBLE) [100]
Variable (DT_DOUBLE) [100,90]
Variable/Adam (DT_DOUBLE) [100,90]
Variable/Adam_1 (DT_DOUBLE) [100,90]
Variable_1 (DT_DOUBLE) [100,90]
Variable_1/Adam (DT_DOUBLE) [100,90]
Variable_1/Adam_1 (DT_DOUBLE) [100,90]
beta1_power (DT_FLOAT) []
beta2_power (DT_FLOAT) []
[[Node: save/RestoreV2_56 = RestoreV2[dtypes=[DT_DOUBLE], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_56/tensor_names, save/RestoreV2_56/shape_and_slices)]]
[[Node: save/RestoreV2_66/_91 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_172_save/RestoreV2_66", tensor_type=DT_DOUBLE, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
Any idea?
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 25 (15 by maintainers)
Hi, sorry for the delay. I thought I’d have time to look at this but apparently I do not! @martinwicke do you know who can look this? (This = a checkpoint from 1.0 can’t be loaded from 1.3, for more details start reading here: https://github.com/tensorflow/tensorflow/issues/7490#issuecomment-332373089)