tensorflow: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912

I am following this tutorial http://www.bitfusion.io/2016/08/31/training-a-bird-classifier-with-tensorflow-and-tflearn/ I assume that training was done but the system was restarted so I can’t verify if the 100 epochs were done. Can you please suggest fixes? Is this a tflearn and tensorflow version mismatch? What can be done?

mona@pascal:~/computer_vision/python_playground$ python infer.py test_images/
bird_african_fish_eagle.jpg          bird_mount_bluebird.jpg              not_a_bird_creativecommons_logo.jpg  
bird_bullocks_oriole.jpg             not_a_bird_airplane.jpg              not_a_bird_stop_sign.jpg             
mona@pascal:~/computer_vision/python_playground$ python infer.py test_images/not_a_bird_stop_sign.jpg 
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: Tesla K40c
major: 3 minor: 5 memoryClockRate (GHz) 0.745
pciBusID 0000:03:00.0
Total memory: 11.92GiB
Free memory: 11.85GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x3771170
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties: 
name: Tesla K40c
major: 3 minor: 5 memoryClockRate (GHz) 0.745
pciBusID 0000:83:00.0
Total memory: 11.92GiB
Free memory: 11.85GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:777] Peer access not supported between device ordinals 0 and 1
I tensorflow/core/common_runtime/gpu/gpu_device.cc:777] Peer access not supported between device ordinals 1 and 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y N 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1:   N Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40c, pci bus id: 0000:03:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K40c, pci bus id: 0000:83:00.0)
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tflearn/summaries.py:46 in get_summary.: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tflearn/summaries.py:46 in get_summary.: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tflearn/helpers/trainer.py:766 in create_summaries.: merge_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.
Instructions for updating:
Please switch to tf.summary.merge.
WARNING:tensorflow:VARIABLES collection name is deprecated, please use GLOBAL_VARIABLES instead; VARIABLES will be removed after 2017-03-02.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tflearn/helpers/trainer.py:130 in __init__.: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40c, pci bus id: 0000:03:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K40c, pci bus id: 0000:83:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40c, pci bus id: 0000:03:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K40c, pci bus id: 0000:83:00.0)
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tflearn/helpers/trainer.py:378 in restore.: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
Traceback (most recent call last):
  File "infer.py", line 44, in <module>
    model.load("bird-classifier.tfl.ckpt-50912")
  File "/usr/local/lib/python2.7/dist-packages/tflearn/models/dnn.py", line 227, in load
    self.trainer.restore(model_file)
  File "/usr/local/lib/python2.7/dist-packages/tflearn/helpers/trainer.py", line 379, in restore
    self.restorer.restore(self.session, model_file)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1388, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 766, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 964, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1014, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1034, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
     [[Node: save_1/RestoreV2_14 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/RestoreV2_14/tensor_names, save_1/RestoreV2_14/shape_and_slices)]]
     [[Node: save_1/RestoreV2_21/_17 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_158_save_1/RestoreV2_21", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op u'save_1/RestoreV2_14', defined at:
  File "infer.py", line 43, in <module>
    model = tflearn.DNN(network, tensorboard_verbose=0, checkpoint_path='bird-classifier.tfl.ckpt')
  File "/usr/local/lib/python2.7/dist-packages/tflearn/models/dnn.py", line 57, in __init__
    session=session)
  File "/usr/local/lib/python2.7/dist-packages/tflearn/helpers/trainer.py", line 125, in __init__
    keep_checkpoint_every_n_hours=keep_checkpoint_every_n_hours)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1000, in __init__
    self.build()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1030, in build
    restore_sequentially=self._restore_sequentially)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 624, in build
    restore_sequentially, reshape)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 361, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 200, in restore_op
    [spec.tensor.dtype])[0])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 441, in restore_v2
    dtypes=dtypes, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): Unsuccessful TensorSliceReader constructor: Failed to find any matching files for bird-classifier.tfl.ckpt-50912
     [[Node: save_1/RestoreV2_14 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/RestoreV2_14/tensor_names, save_1/RestoreV2_14/shape_and_slices)]]
     [[Node: save_1/RestoreV2_21/_17 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_158_save_1/RestoreV2_21", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

[1]+  Killed                  python2 infer.py ${f} 2> /dev/null
mona@pascal:~/computer_vision/python_playground$ ls *50912(
-bash: syntax error near unexpected token `('
mona@pascal:~/computer_vision/python_playground$ ls *50912*
bird-classifier.tfl.ckpt-50912.data-00000-of-00001  bird-classifier.tfl.ckpt-50912.index  bird-classifier.tfl.ckpt-50912.meta

Here are the dumped training files I have http://pastebin.com/9RF58yBB

Environment info

Operating System:

$ uname -a ; lsb_release -a 
Linux pascal 3.13.0-62-generic #102-Ubuntu SMP Tue Aug 11 14:29:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.5 LTS
Release:	14.04
Codename:	trusty

Installed version of CUDA and cuDNN: (please attach the output of ls -l /path/to/cuda/lib/libcud*):

$ ls /usr/local/cuda-8.0/lib64/
libcublas_device.a   libcudnn.so.5.1.10   libcuinj64.so.8.0.44   libcusparse.so.8.0.44  libnppicom.so         libnppim.so          libnppisu.so.8.0.44  libnvgraph.so.8.0            libOpenCL.so
libcublas.so         libcudnn_static.a    libculibos.a           libcusparse_static.a   libnppicom.so.8.0     libnppim.so.8.0      libnppitc.so         libnvgraph.so.8.0.44         libOpenCL.so.1
libcublas.so.8.0     libcufft.so          libcurand.so           libnppc.so             libnppicom.so.8.0.44  libnppim.so.8.0.44   libnppitc.so.8.0     libnvgraph_static.a          libOpenCL.so.1.0
libcublas.so.8.0.45  libcufft.so.8.0      libcurand.so.8.0       libnppc.so.8.0         libnppidei.so         libnppi.so           libnppitc.so.8.0.44  libnvrtc-builtins.so         libOpenCL.so.1.0.0
libcublas_static.a   libcufft.so.8.0.44   libcurand.so.8.0.44    libnppc.so.8.0.44      libnppidei.so.8.0     libnppi.so.8.0       libnpps.so           libnvrtc-builtins.so.8.0     stubs
libcudadevrt.a       libcufft_static.a    libcurand_static.a     libnppc_static.a       libnppidei.so.8.0.44  libnppi.so.8.0.44    libnpps.so.8.0       libnvrtc-builtins.so.8.0.44
libcudart.so         libcufftw.so         libcusolver.so         libnppial.so           libnppif.so           libnppi_static.a     libnpps.so.8.0.44    libnvrtc.so
libcudart.so.8.0     libcufftw.so.8.0     libcusolver.so.8.0     libnppial.so.8.0       libnppif.so.8.0       libnppist.so         libnpps_static.a     libnvrtc.so.8.0
libcudart.so.8.0.44  libcufftw.so.8.0.44  libcusolver.so.8.0.44  libnppial.so.8.0.44    libnppif.so.8.0.44    libnppist.so.8.0     libnvblas.so         libnvrtc.so.8.0.44
libcudart_static.a   libcufftw_static.a   libcusolver_static.a   libnppicc.so           libnppig.so           libnppist.so.8.0.44  libnvblas.so.8.0     libnvToolsExt.so
libcudnn.so          libcuinj64.so        libcusparse.so         libnppicc.so.8.0       libnppig.so.8.0       libnppisu.so         libnvblas.so.8.0.44  libnvToolsExt.so.1
libcudnn.so.5        libcuinj64.so.8.0    libcusparse.so.8.0     libnppicc.so.8.0.44    libnppig.so.8.0.44    libnppisu.so.8.0     libnvgraph.so        libnvToolsExt.so.1.0.0

If installed from binary pip package, provide:

A link to the pip package you installed:
The output from python -c "import tensorflow; print(tensorflow.__version__)".

$ python -c "import tensorflow; print(tensorflow.__version__)"
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
0.12.1

If installed from source, provide

The commit hash (git rev-parse HEAD)

mona@pascal:~/tf/tensorflow$ git rev-parse HEAD
156da397dc2e354baeac10804c5e9c1b3af8b7eb

The output of bazel version

$ bazel version
...........................
Build label: 0.4.3
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Dec 22 12:31:25 2016 (1482409885)
Build timestamp: 1482409885
Build timestamp as int: 1482409885

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 29 (3 by maintainers)

Most upvoted comments

Can you try changing this line in your code:

model.load("bird-classifier.tfl.ckpt-50912")

…to the following:

model.load("./bird-classifier.tfl.ckpt-50912")

…and let us know if that works?

+14

mrry on Feb 17, 2017

I had the identical issue with r1.0.0 and adding the “./” in front of the file name resolved the issue for me.

redbullpeter on Feb 22, 2017

i had solved by this way #fine_tune_checkpoint: “/root/code/tensorflow/models/training/model.ckpt-#####” from_detection_checkpoint: false please check your config

watchpoints on Aug 21, 2018

Maybe a missing ./ in front of ssd_mobilenet_v1_coco_2017_11_17/model.ckpt. This issue was already answered here: https://stackoverflow.com/questions/42260167/unsuccessful-tensorslicereader-constructor-failed-to-find-any-matching-files-fo

chriskraus0 on Aug 6, 2018

@monajalal Did adding the ./ work for you?

mrry on Feb 22, 2017

I tried Traning in Google colab python train.py --train_dir=training/ --pipeline_config_path=ssd_mobilenet_v2_quantized_300x300_coco.config but I am getting following error

WARNING:tensorflow:From train.py:56: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From train.py:56: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

WARNING:tensorflow:From train.py:185: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/absl/app.py:251: main (from main) is deprecated and will be removed in a future version. Instructions for updating: Use object_detection/model_main.py. W1101 10:00:07.914612 140433468364672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/absl/app.py:251: main (from main) is deprecated and will be removed in a future version. Instructions for updating: Use object_detection/model_main.py. WARNING:tensorflow:From train.py:91: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W1101 10:00:07.914868 140433468364672 module_wrapper.py:139] From train.py:91: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

WARNING:tensorflow:From train.py:96: The name tf.gfile.Copy is deprecated. Please use tf.io.gfile.copy instead.

W1101 10:00:07.919008 140433468364672 module_wrapper.py:139] From train.py:96: The name tf.gfile.Copy is deprecated. Please use tf.io.gfile.copy instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/object_detection/legacy/trainer.py:265: create_global_step (from tf_slim.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.create_global_step W1101 10:00:07.932987 140433468364672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/object_detection/legacy/trainer.py:265: create_global_step (from tf_slim.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.create_global_step INFO:tensorflow:Reading unweighted datasets: [‘train.record’] I1101 10:00:07.952460 140433468364672 dataset_builder.py:148] Reading unweighted datasets: [‘train.record’] INFO:tensorflow:Reading record datasets for input file: [‘train.record’] I1101 10:00:07.953956 140433468364672 dataset_builder.py:77] Reading record datasets for input file: [‘train.record’] INFO:tensorflow:Number of filenames to read: 1 I1101 10:00:07.954146 140433468364672 dataset_builder.py:78] Number of filenames to read: 1 WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards. W1101 10:00:07.954262 140433468364672 dataset_builder.py:86] num_readers has been reduced to 1 to match input file shards. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/object_detection/builders/dataset_builder.py:103: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_determinstic. W1101 10:00:07.962062 140433468364672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/object_detection/builders/dataset_builder.py:103: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_determinstic. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/object_detection/builders/dataset_builder.py:222: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.map() W1101 10:00:07.992240 140433468364672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/object_detection/builders/dataset_builder.py:222: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.map() WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/object_detection/builders/dataset_builder.py:48: DatasetV1.make_initializable_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use for ... in dataset: to iterate over a dataset. If using tf.estimator, return the Dataset object directly from your input function. As a last resort, you can use tf.compat.v1.data.make_initializable_iterator(dataset). W1101 10:00:10.920446 140433468364672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/object_detection/builders/dataset_builder.py:48: DatasetV1.make_initializable_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use for ... in dataset: to iterate over a dataset. If using tf.estimator, return the Dataset object directly from your input function. As a last resort, you can use tf.compat.v1.data.make_initializable_iterator(dataset). WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/object_detection/core/preprocessor.py:199: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: seed2 arg is deprecated.Use sample_distorted_bounding_box_v2 instead. W1101 10:00:10.988368 140433468364672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/object_detection/core/preprocessor.py:199: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: seed2 arg is deprecated.Use sample_distorted_bounding_box_v2 instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/object_detection/core/box_list_ops.py:234: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where W1101 10:00:11.013556 140433468364672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/object_detection/core/box_list_ops.py:234: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/object_detection/core/batcher.py:101: batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.batch(batch_size) (or padded_batch(...) if dynamic_pad=True). W1101 10:00:11.954241 140433468364672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/object_detection/core/batcher.py:101: batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.batch(batch_size) (or padded_batch(...) if dynamic_pad=True). WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/input.py:752: QueueRunner.init (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. W1101 10:00:11.958855 140433468364672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/input.py:752: QueueRunner.init (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/input.py:752: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. W1101 10:00:11.960053 140433468364672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/input.py:752: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tf_slim-1.1.0-py3.6.egg/tf_slim/layers/layers.py:2802: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use layer.__call__ method instead. W1101 10:00:12.111274 140433468364672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tf_slim-1.1.0-py3.6.egg/tf_slim/layers/layers.py:2802: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use layer.__call__ method instead. INFO:tensorflow:depth of additional conv before box predictor: 0 I1101 10:00:16.065051 140433468364672 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I1101 10:00:16.101743 140433468364672 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I1101 10:00:16.137190 140433468364672 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I1101 10:00:16.173719 140433468364672 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I1101 10:00:16.211551 140433468364672 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I1101 10:00:16.247035 140433468364672 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/rmsprop.py:119: calling Ones.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor W1101 10:00:18.863176 140433468364672 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/rmsprop.py:119: calling Ones.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/moving_averages.py:433: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. W1101 10:00:21.508541 140433468364672 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/moving_averages.py:433: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. Traceback (most recent call last): File “train.py”, line 185, in <module> tf.app.run() File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py”, line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File “/usr/local/lib/python3.6/dist-packages/absl/app.py”, line 300, in run _run_main(main, args) File “/usr/local/lib/python3.6/dist-packages/absl/app.py”, line 251, in _run_main sys.exit(main(argv)) File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py”, line 324, in new_func return func(*args, **kwargs) File “train.py”, line 181, in main graph_hook_fn=graph_rewriter_fn) File “/usr/local/lib/python3.6/dist-packages/object_detection/legacy/trainer.py”, line 396, in train include_global_step=False)) File “/usr/local/lib/python3.6/dist-packages/object_detection/utils/variables_helper.py”, line 138, in get_variables_available_in_checkpoint ckpt_reader = tf.train.NewCheckpointReader(checkpoint_path) File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/pywrap_tensorflow_internal.py”, line 873, in NewCheckpointReader return CheckpointReader(compat.as_bytes(filepattern)) File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/pywrap_tensorflow_internal.py”, line 885, in init this = _pywrap_tensorflow_internal.new_CheckpointReader(filename) tensorflow.python.framework.errors_impl.InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /content/models-master/research/object_detection/ssd_inception_v2_coco_2018_01_28/model.ckpt: Not found: /content/models-master/research/object_detection/ssd_inception_v2_coco_2018_01_28; No such file or directory

Jayanth1812 on Nov 1, 2020

yes it is clear it not found this file, please check you tree folder, you have to download it , by default it does’nt not exist in object_detection folder

leccyril on Aug 6, 2018

Restoring a checkpoint failed for me when the path of the checkpoint contained symbols like [ or ,. Removing these characters solved the error.

gcucurull on Jul 12, 2018

i don’t know why but sometimes we have to provide absolute path for the model ckpt

leccyril on Jul 5, 2018

Update: Instead of sofamodels/model.ckpt or ./sofamodels/model.ckpt we have to mention the entire path gs://bucketname/sofamodels/model.ckpt.

JP-Reddy on Oct 11, 2017

Yes, I do think so 😃. @rohan100jain fixed this bug recently, so it should work as expected if you use a nightly build.

mrry on Mar 8, 2017

@monajalal No, I am running a repurposed version of this TensorKart

redbullpeter on Feb 22, 2017