tensorflow: tf.contrib.summary.image() fails silently and inconsistently

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No
  • TensorFlow installed from (source or binary): Source
  • TensorFlow version (use command below): b’v1.12.0-5845-g764109a352’ 1.12.0
  • Python version: 3.6.7
  • Bazel version (if compiling from source): Invocation ID: 42251854-036f-415c-8a52-76aac8520ea0 Build label: 0.21.0 Build time: Wed Dec 19 12:58:44 2018 (1545224324)
  • GCC/Compiler version (if compiling from source): gcc (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0
  • CUDA/cuDNN version: Cuda compilation tools, release 10.0, V10.0.130
  • GPU model and memory: GTX 1060 Max Q, 6gb VRAM

Describe the current behavior I am running into a very weird problem where sometimes images will be logged in tensorboard / the event file, but most of the time they are not.

The issue with this is that I can’t get consistent behaviour. One time I will run it and it will log normally, and then without changing any code, the next time it won’t. And it is really hard to find out anything when tensorboard --inspect shows nothing under -image either.

I should mention that the input data is not always the same, although it is always taken from the same state space (2D game environment with finite number of possible states).

Describe the expected behavior If it was not able to log my images, the function should at the very least provide some information to the user as to why it was not able to do so.

Code to reproduce the issue This is not possible, as I can not consistently reproduce it myself. It is seemingly completely random, and provides no log, error or warning information.

Other info / logs As stated above, there are no logs of any sort.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 20 (7 by maintainers)

Most upvoted comments

From what I understand, if you take a look at tf.summary.image, it details that it does preprocessing to the image before logging it. I assume the tf.contrib.summary.image does the same. It’s obvious that there are problems with this function, so I would try to log some other image data to make sure your input is correct, and make sure the batch sizes are the same for all data.