tensorflow: tf.keras multi input models don't work when using tf.data.Dataset

I think I discovered the problem fro my situation. The problem was I am using the standalone Keras. Not the one imported from tendorflow. So the new features of feeding the iterator directly to model.fit() is valid only when you are usingtf.Kerasnot the standalone Keras.

@JanRuettinger @ricoms Sorry for the delayed response.

I drafted up a toy example using MNIST in order to train a model with two inputs and two outputs. The model is simply two identical models fused together, which takes in two copies of the MNIST data (two inputs) and outputs a prediction for each (two outputs). You can adapt this to more complex models and input pipelines.

Note: This is still using tf-nightly-gpu version 1.12.0-dev20180918. I assume this will work in tensorflow 1.12 and above.

batch_size = 512

# -- Data Setup -- #
(x_train, y_train),(x_test, y_test) = tf.keras.datasets.mnist.load_data()
y_train = tf.keras.utils.to_categorical(y_train)
x_train, x_test = x_train / 255.0, x_test / 255.0
# Create two inputs and two outputs (for demonstration)
x_train1 = x_train2 = x_train
y_train1 = y_train2 = y_train

# -- Dataset API -- #
# Create a Dataset for multiple inputs and Dataset for multiple outputs
input_set = tf.data.Dataset.from_tensor_slices((x_train1, x_train2))
output_set = tf.data.Dataset.from_tensor_slices((y_train1, y_train2))
# Create Dataset pipeline
input_set = input_set.batch(batch_size).repeat()
output_set = output_set.batch(batch_size).repeat()
# Group the input and output dataset
dataset = tf.data.Dataset.zip((input_set, output_set))
# Initialize the iterator to be passed to the model.fit() function
data_iter = dataset.make_one_shot_iterator()

# -- Model Definition -- #
# Multiple Inputs
input1 = tf.keras.layers.Input(shape=(28,28))
input2 = tf.keras.layers.Input(shape=(28,28))
# Input 1 Pathway
x1 = tf.keras.layers.Flatten()(input1)
x1 = tf.keras.layers.Dense(512, activation=tf.nn.relu)(x1)
x1 = tf.keras.layers.Dropout(0.2)(x1)
# Input 2 Pathway
x2 = tf.keras.layers.Flatten()(input2)
x2 = tf.keras.layers.Dense(512, activation=tf.nn.relu)(x2)
x2 = tf.keras.layers.Dropout(0.2)(x2)
# Multiple Outputs
output1 = tf.keras.layers.Dense(10, activation=tf.nn.softmax)(x1)
output2 = tf.keras.layers.Dense(10, activation=tf.nn.softmax)(x2)
# Create Model
model = tf.keras.models.Model(inputs=[input1, input2], outputs=[output1, output2])
# Compile
model.compile(optimizer='adam', loss='categorical_crossentropy')

# -- Train -- #
model.fit(data_iter, steps_per_epoch=len(x_train)//batch_size, epochs=5)

Update: As @jashshopin mentions below, the dataset object can be passed directly to model.fit() if you have no need for an iterator.

Hi, @was84san. As you mentioned, I am using tf.kera. But the problem still exists. Do you have any idea? Thanks!

I am not finding documentation for feeding models with multiple inputs with different dimensions with tf.data.
The above exchange leaves me still struggling for an understanding on feeding such models. May I asked for clarification?

print(f"tensoflow.__version__ = {tensorflow.__version__}")
# tensoflow.__version__ = 2.1.0-rc2

# A toy keras model with 2 inputs of different size
input_1 = tensorflow.keras.layers.Input(name='input_1', shape=(2,), dtype=numpy.float32)
input_2 = tensorflow.keras.layers.Input(name='input_2', shape=(3,), dtype=numpy.float32)
output = tensorflow.keras.layers.Concatenate(name='output_1')([input_1, input_2])
toy_model = tensorflow.keras.Model(inputs=[input_1, input_2], outputs=[output])
toy_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# in memory data, 2 samples for input_1
input_1_sample_1 = numpy.asarray ( [2,2], dtype=numpy.float32 )
input_1_sample_2 = numpy.asarray ( [22,22], dtype=numpy.float32 )
input_1_data = numpy.asarray( [ input_1_sample_1, input_1_sample_2 ] )
print(f"input_1_data.shape = {input_1_data.shape}")
# input_1_data.shape = (2, 2)

# in memory data, 2 samples for input_2
input_2_sample_1 = numpy.asarray ( [3,3,3], dtype=numpy.float32 )
input_2_sample_2 = numpy.asarray ( [33,33,33], dtype=numpy.float32 )
input_2_data = numpy.asarray( [ input_2_sample_1, input_2_sample_2 ] )
print(f"input_2_data.shape = {input_2_data.shape}")
# input_2_data.shape = (2, 3)

# in memory data, 2 samples for output 1
output_1_sample_1 = numpy.asarray( [2,2,3,3,3], dtype=numpy.float32 )
output_1_sample_2 = numpy.asarray( [22,22,33,33,33], dtype=numpy.float32 )
output_1_data = numpy.asarray( [ output_1_sample_1, output_1_sample_2], dtype=numpy.float32 )
print(f"output_1_data.shape = {output_1_data.shape}")
# output_1_data.shape = (2, 5)

def toy_generator_list():
    while True:
        yield [input_1_data, input_2_data], output_1_data, []

I can use the generator directly, but my goal is to move the generator to a full tf.data pipleline, but I am missing something fundamental to get started.

This works, but does not use tf.data:

toy_model.fit(x=toy_generator_list(), steps_per_epoch=3, epochs=2)

The following as close to a solution I have gotten to, but it fails

toy_dataset_from_generator = tensorflow.data.Dataset.from_generator(toy_generator_list, \
    output_types=(tensorflow.float32, tensorflow.float32, tensorflow.float32), \
        output_shapes=(([2,2],[2,3]), [2,5]) )


toy_model.fit(x=toy_dataset_from_generator, steps_per_epoch=3, epochs=2) 

Generates error

ValueError: The two structures don't have the same sequence length. Input structure has length 2, while the shallow structure has length 3.

I know that my request smells like “a request for help”, it is, but please interpret it as a request for improved documentation. Stack overflow does not have anything on multiple inputs of different shapes.

btw:

• The real model input is an image (255, 255, 3) and a document type (20,) with output 1-hot(40, 60) into a ctc.
• The ideal tf.data chain would cache the preliminary processing, then augment this cache version for delivery to model.fit, where the model is fit across a network of servers.

@ Igeiger, I tried to pass multiple inputs as a list of tf.dataset api to model fit directly, like this model.fit ( [dataset1_iterator, dataset2_iterator] , .....)

then I got this error

 /home/wassan/tensorflow/venv/lib/python2.7/site- 
 packages/tensorflow/python/keras/engine/training.pyc in _standardize_user_data(self, x, y, sample_weight, cla$s_weight, batch_size, check_steps, steps_name, steps, validation_split)
    990         x, y, sample_weight = next_element
    991     x, y, sample_weights = self._standardize_weights(x, y, sample_weight,
--> 992                                                      class_weight, batch_size)
    993     return x, y, sample_weights
    994 

/home/wassan/tensorflow/venv/lib/python2.7/site-packages/tensorflow/python/keras/engine/training.pyc in _standardize_weights(self, x, y, sample_weight, class$weight, batch_size)
   1115         feed_input_shapes,
   1116         check_batch_axis=False,  # Don't enforce the batch size.
-> 1117         exception_prefix='input')
   1118 
   1119     if y is not None:

/home/wassan/tensorflow/venv/lib/python2.7/site-packages/tensorflow/python/keras/engine/training_utils.pyc in standardize_input_data(data, names, shapes, che$k_batch_axis, exception_prefix)
    282     data = data.values if data.__class__.__name__ == 'DataFrame' else data
    283     data = [data]
--> 284   data = [standardize_single_array(x) for x in data]
    285 
    286   if len(data) != len(names):

/home/wassan/tensorflow/venv/lib/python2.7/site-packages/tensorflow/python/keras/engine/training_utils.pyc in standardize_single_array(x)
    216   if x is None:
    217     return None
--> 218   if x.shape is not None and len(x.shape) == 1:
    219     if tensor_util.is_tensor(x):
    220       return array_ops.expand_dims(x, axis=1)

AttributeError: 'PrefetchDataset' object has no attribute 'shape

And this is with tensorflow 1.12, so how you can pass multiple input using tf.dataset api with model fit not with model.fit_generator? `

@jashshopin Thanks for pointing that out, apparently you can pass the zipped dataset directly into model.fit(). The example should still work for those who might want to use a one-shot iterator or initializable iterator as well.

@gabrielibagon Could you post a snippet how you got a nested dataset iterator with multiple inputs working?

I think I discovered the problem fro my situation. The problem was I am using the standalone Keras. Not the one imported from tendorflow. So the new features of feeding the iterator directly to model.fit() is valid only when you are usingtf.Kerasnot the standalone Keras.

Thx, my problem solved! Just have changed import keras import tensorflow as tf to import tensorflow as tf from tensorflow import keras

This still seems broken to me (in tensorflow 2.0.0-rc0). See this snippet:

import tensorflow as tf
from tensorflow import keras

inputs = [keras.Input((1,), name="a"), keras.Input((1,), name="b")]
outputs = inputs[0] + inputs[1]
model = keras.Model(inputs=inputs, outputs=outputs)

list_input = [tf.zeros((10, 1)), tf.ones((10, 1))]
dict_input = {"a": tf.zeros((10, 1)), "b": tf.ones((10, 1))}

print(model.predict(list_input))
print(model.predict(dict_input))
print(model.predict(tf.data.Dataset.from_tensors(dict_input)))

# error here
print(model.predict(tf.data.Dataset.from_tensors(list_input)))

which gives

ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [<tf.Tensor: id=47, shape=(2, 10, 1), dtype=float32, numpy=
array([[[0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.]],...

I can confirm this works in tensorflow 2.0.0-rc0. Multiple input and output, even without all the zipping:

(train_images, _), (test_images, _) = tf.keras.datasets.mnist.load_data()

ds = tf.data.Dataset.from_tensor_slices( ((train_images, dummydata), train_images) )
ds.shuffle(TRAIN_BUF).repeat().batch(BATCH_SIZE)

model.fit(train_dataset, steps_per_epoch=n_trainsamples//BATCH_SIZE)

@johngrabner The problem in your code is that your output_type and output_shape definitions differ. Changing the output_type to ((tensorflow.float32, tensorflow.float32), tensorflow.float32) should to the trick.

For the sake of completeness, here is a minimal example of a dataset that expects two inputs (shapes (1,32) and (1,128)):

import tensorflow as tf
import numpy as np

def random_generator():
    for i in range(100):
        x1, x2, y = np.random.random((1,32)), np.random.random((1,128)), np.random.random((1,1))
        yield (x1, x2), y
        
toy_dataset = tf.data.Dataset.from_generator(
    random_generator,
    output_types=((tf.float32, tf.float32), tf.float32),
    output_shapes=(((1,32), (1,128)), (1,1))
)

@lgeiger what about using dictionaries as targets? https://github.com/tensorflow/tensorflow/issues/25299#issue-404556539