keras: TensorBoard callback does not create histograms when a generator is used to create validation data

Closing as this is resolved

A relatively simple fix is to “fill in” the validation_data property before the TensorBoard on_epoch_end hook is called by inheriting TensorBoard in a wrapper like below. Obviously the way you fill in validation_data is specific to your problem. You can just replace your TensorBoard callback with TensorBoardWrapper, pass the batch_gen and nb_steps arguments, and then all of the same arguments as `Tensorboard. Unfortunately this also means that if you are using a generator for validation, it will get called once in the wrapper and then again for validation. If you can afford to keep your data in memory, the below solution could be moved into the on_train_begin hook.

class TensorBoardWrapper(TensorBoard):
    '''Sets the self.validation_data property for use with TensorBoard callback.'''

    def __init__(self, batch_gen, nb_steps, **kwargs):
        super().__init__(**kwargs)
        self.batch_gen = batch_gen # The generator.
        self.nb_steps = nb_steps     # Number of times to call next() on the generator.

    def on_epoch_end(self, epoch, logs):
        # Fill in the `validation_data` property. Obviously this is specific to how your generator works.
        # Below is an example that yields images and classification tags.
        # After it's filled in, the regular on_epoch_end method has access to the validation_data.
        imgs, tags = None, None
        for s in range(self.nb_steps):
            ib, tb = next(self.batch_gen)
            if imgs is None and tags is None:
                imgs = np.zeros((self.nb_steps * ib.shape[0], *ib.shape[1:]), dtype=np.float32)
                tags = np.zeros((self.nb_steps * tb.shape[0], *tb.shape[1:]), dtype=np.uint8)
            imgs[s * ib.shape[0]:(s + 1) * ib.shape[0]] = ib
            tags[s * tb.shape[0]:(s + 1) * tb.shape[0]] = tb
        self.validation_data = [imgs, tags, np.ones(imgs.shape[0]), 0.0]
        return super().on_epoch_end(epoch, logs)


...

callbacks = [TensorBoardWrapper(gen_val, nb_steps=5, log_dir=self.cfg['cpdir'], histogram_freq=1,
                               batch_size=32, write_graph=False, write_grads=True)]
...

This is still an issue for me in 2.0.5. It writes out scalars and graphs, but no histograms, or distributions.

does the pull request will be merged? I’m having the same use case

Would be really nice to get this fixed. Having the same problem in keras 1.2.2 - 2.0.4.

same problem here!

Using

class TensorBoardWrapper(TensorBoard):
    '''Sets the self.validation_data property for use with TensorBoard callback.'''

    def __init__(self, batch_gen, nb_steps, b_size, **kwargs):
        super(TensorBoardWrapper, self).__init__(**kwargs)
        self.batch_gen = batch_gen # The generator.
        self.nb_steps = nb_steps   # Number of times to call next() on the generator.
        #self.batch_size = b_size

    def on_epoch_end(self, epoch, logs):
        # Fill in the `validation_data` property. Obviously this is specific to how your generator works.
        # Below is an example that yields images and classification tags.
        # After it's filled in, the regular on_epoch_end method has access to the validation_data.
        imgs, tags = None, None
        for s in range(self.nb_steps):
            ib, tb = next(self.batch_gen)
            if imgs is None and tags is None:
                imgs = np.zeros(((self.nb_steps * self.batch_size,) + ib.shape[1:]), dtype=np.float32)
                tags = np.zeros(((self.nb_steps * self.batch_size,) + tb.shape[1:]), dtype=np.uint8)
            imgs[s * ib.shape[0]:(s + 1) * ib.shape[0]] = ib
            tags[s * tb.shape[0]:(s + 1) * tb.shape[0]] = tb
        
        self.validation_data = [imgs, tags, np.ones(imgs.shape[0])]
              
        return super(TensorBoardWrapper, self).on_epoch_end(epoch, logs)

and

TensorBoardWrapper(self.val_generator2, ceil(self.val_dataset_size / self.batch_size), self.batch_size, log_dir="{}/{}".format(self.logs_dir, time()), histogram_freq=1, batch_size=self.batch_size)

worked for me. Notice that initialization of imgs and tags uses batch_size ( imgs = np.zeros(((self.nb_steps * self.batch_size,) + ib.shape[1:]), dtype=np.float32)) instead of the first element of the batch’s shape (imgs = np.zeros(((self.nb_steps * ib.shape[0],) + ib.shape[1:]), dtype=np.float32)).

This is because if total_batches % batch_size != 0 the first call to next(self.batch_gen) will return a batch who’s shape’s first element is not equal to the batch size, resulting in the same broadcast shape error @NTNguyen13 reported.

Then I got an AssertionError in loc 884 from Keras’ callbacks.py. According to that loc validation_data must have at most three elements just like that tensors array.

I fixed that by changing self.validation_data = [imgs, tags, np.ones(imgs.shape[0]), 0.0] to self.validation_data = [imgs, tags, np.ones(imgs.shape[0])]. I kept the third element as np.ones(imgs.shape[0]) because my generator only outputs images and labels.

Remember to use a generator that supports multi-threading or use two instances of the same generator to avoid getting a ValueError: generator already executing. I used two instances for a quick fix.

I’m using keras 2.1.4 and tensorflow-gpu 1.4.1 on one NVIDIA Titan Xp with CUDA 8 and I haven’t run into any memory issues.

If i understand correctly the solution @alexklibisz suggested, we still have to load the entire validation suite into memory. Is there a way to view the histograms on the entire validation set without loading all in once to memory?

I managed to get @juiceboxjoe s version running, but I had to include some of the sections he removed ( self.batch_size = b_size, self.validation_data = [imgs, tags, np.ones(imgs.shape[0]), 0.0] )

I’m using keras 2.2.4 based on tensorflow-gpu version 1.13.1

My full Wrapper:

class TensorBoardWrapper(keras.callbacks.TensorBoard):
    '''Sets the self.validation_data property for use with TensorBoard callback.'''

    def __init__(self, batch_gen, nb_steps, b_size, **kwargs):
        super(TensorBoardWrapper, self).__init__(**kwargs)
        self.batch_gen = batch_gen # The generator.
        self.nb_steps = nb_steps   # Number of times to call next() on the generator.
        self.batch_size = b_size

    def on_epoch_end(self, epoch, logs):
        # Fill in the `validation_data` property. Obviously this is specific to how your generator works.
        # Below is an example that yields images and classification tags.
        # After it's filled in, the regular on_epoch_end method has access to the validation_data.
        imgs, tags = None, None
        for s in range(self.nb_steps):
            ib, tb = next(self.batch_gen)
            if imgs is None and tags is None:
                imgs = np.zeros(((self.nb_steps * self.batch_size,) + ib.shape[1:]), dtype=np.float32)
                tags = np.zeros(((self.nb_steps * self.batch_size,) + tb.shape[1:]), dtype=np.float32)
            imgs[s * ib.shape[0]:(s + 1) * ib.shape[0]] = ib
            tags[s * tb.shape[0]:(s + 1) * tb.shape[0]] = tb
        
        self.validation_data = [imgs, tags, np.ones(imgs.shape[0]), 0.0]
              
        return super(TensorBoardWrapper, self).on_epoch_end(epoch, logs)

Called like this:

tBCallback = TensorBoardWrapper(test_it, math.ceil(image_count_test/config.batch_size), config.batch_size,
                                log_dir=model_dir_path, histogram_freq=5, write_graph=True, write_images=True, write_grads=True)

I used alexklibisz’s code above and it got me some TensorBoard functionality working using validation generators (ie. TensorBoard scalars). But TensorBoard Images were still blank ie.

does anyone know a way to get Images working with validation generators?

Thanks a lot for the code snippet! Here’s the Python2 copy & paste version for lazy people like me:

class TensorBoardWrapper(ks.callbacks.TensorBoard):
    '''Sets the self.validation_data property for use with TensorBoard callback.'''

    def __init__(self, batch_gen, nb_steps, **kwargs):
        super(TensorBoardWrapper, self).__init__(**kwargs)
        self.batch_gen = batch_gen # The generator.
        self.nb_steps = nb_steps   # Number of times to call next() on the generator.

    def on_epoch_end(self, epoch, logs):
        # Fill in the `validation_data` property. Obviously this is specific to how your generator works.
        # Below is an example that yields images and classification tags.
        # After it's filled in, the regular on_epoch_end method has access to the validation_data.
        imgs, tags = None, None
        for s in range(self.nb_steps):
            ib, tb = next(self.batch_gen)
            if imgs is None and tags is None:
                imgs = np.zeros(((self.nb_steps * ib.shape[0],) + ib.shape[1:]), dtype=np.float32)
                tags = np.zeros(((self.nb_steps * tb.shape[0],) + tb.shape[1:]), dtype=np.uint8)
            imgs[s * ib.shape[0]:(s + 1) * ib.shape[0]] = ib
            tags[s * tb.shape[0]:(s + 1) * tb.shape[0]] = tb
        self.validation_data = [imgs, tags, np.ones(imgs.shape[0]), 0.0]
        return super(TensorBoardWrapper, self).on_epoch_end(epoch, logs)