tensorflow: tf.keras.estimator.model_to_estimator fails with pre-trained models

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 7 x64
TensorFlow installed from (source or binary): binary (pip)
TensorFlow version (use command below): 1.7.0 (cpu)
Python version: 3.6
Bazel version (if compiling from source): n.a.
GCC/Compiler version (if compiling from source): n.a.
CUDA/cuDNN version: n.a.
GPU model and memory: n.a.
Exact command to reproduce:

Describe the problem

Following the example code for Creating Estimators from Keras models, I encountered a problem: the example code runs fine when weights=None is used in the initialization of the InceptionV3 object. But when I change it to weights=imagenet the call to model_to_estimator(...) fails:

FailedPreconditionError: Attempting to use uninitialized value batch_normalization_100/beta [[Node: _retval_batch_normalization_100/beta_0_0 = _RetvalT=DT_FLOAT, index=0, _device=“/job:localhost/replica:0/task:0/device:CPU:0”]]

This looks like the weights of the model are not correctly initialized. But strangely I can use the keras model to obtain predictions, which should mean that the model is correctly initialized.

As far as I know there are no information in the documentation that pre-trained models (i.e. models with weights set to anything different than None) are not supported by this function, or that one has to change the workflow somehow. Therefore, this is either a bug in tensorflow or a case of a misleading documentation.

Source code / logs

# Instantiate a Keras inception v3 model.
keras_inception_v3 = tf.keras.applications.inception_v3.InceptionV3(weights="imagenet")
# Compile model with the optimizer, loss, and metrics you'd like to train with.
keras_inception_v3.compile(optimizer=tf.keras.optimizers.SGD(lr=0.0001, momentum=0.9),
                          loss='categorical_crossentropy',
                          metric='accuracy')
# Create an Estimator from the compiled Keras model. Note the initial model
# state of the keras model is preserved in the created Estimator.
est_inception_v3 = tf.keras.estimator.model_to_estimator(keras_model=keras_inception_v3)

# Treat the derived Estimator as you would with any other Estimator.
# First, recover the input name(s) of Keras model, so we can use them as the
# feature column name(s) of the Estimator input function:
keras_inception_v3.input_names  # print out: ['input_1']
# Once we have the input name(s), we can create the input function, for example,
# for input(s) in the format of numpy ndarray:
train_input_fn = tf.estimator.inputs.numpy_input_fn(
    x={"input_1": train_data},
    y=train_labels,
    num_epochs=1,
    shuffle=False)
# To train, we call Estimator's train function:
est_inception_v3.train(input_fn=train_input_fn, steps=2000)

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 9
Comments: 16 (8 by maintainers)

Most upvoted comments

It appears the problem might be that the model is not trained on the the layers from the tensorflow version of Keras. I found that when building a model using from keras.layers import ... then this flags an error. However, using layers in the following form:

from tensorflow import keras

inpt = keras.layers.Input(shape = (...))
x = keras.layers.Convolution2D(...)(inpt)

makes it work.

JosephPB on Jun 6, 2018

@yifeif Generally seems that we are having many problems with tf.keras.estimator.model_to_estimator. What is the problem? Is that is not used internally so frequently in Google? Poor test coverage? What else?

bhack on Apr 6, 2018

from tensorflow.keras.callbacks import TensorBoard, EarlyStopping

I imagine having the same problem but this did not fixed mine

Error:

FailedPreconditionError: Error while reading resource variable Adam_38/lr from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/Adam_38/lr/class tensorflow::Var does not exist. [[{{node training_34/Adam/ReadVariableOp_1}} = ReadVariableOpdtype=DT_FLOAT, _device=“/job:localhost/replica:0/task:0/device:CPU:0”]]

Source code:

tbCallBack =keras.callbacks.TensorBoard(log_dir=name, histogram_freq=0, batch_size=32, write_graph=True, write_grads=False, write_images=False, embeddings_freq=0, embeddings_layer_names=None, embeddings_metadata=None, embeddings_data=None, update_freq='epoch')

    model = tf.keras.models.Sequential([
        tf.keras.layers.Flatten(input_shape=(60,60,)),
        tf.keras.layers.Dropout(0.2)])
    for dense in hiddenLayerStructure:
        model.add(tf.keras.layers.Dense(dense, activation=tf.nn.relu))
    model.add(tf.keras.layers.Dense(2, activation=tf.nn.softmax))
    model.compile(optimizer=optimizer_,loss=loss_,metrics=['accuracy'])
    tbCallBack.set_model(model)
    model.fit(x, y,validation_data=([x], [y]), epochs=epochs_,verbose=2)

Already tried :

Session variable initialization
Using callbacks=[tbCallBack]) instead of tbCallBack.set_model(model)
Using other callbacks works perfectly
tensorflow 1.12.0
keras 2.2.4
Reinstalling both tensorflow and keras using pip
win 10 64
python 3.6

i3130002 on Dec 26, 2018

@JosephPB thanks yous answer

from tensorflow.keras.callbacks import TensorBoard, EarlyStopping

makes it work.

aojue1109 on Sep 19, 2018

Reopen the issue and wait for further comments from @yifeif

qlzh727 on Apr 10, 2018

@qlzh727 When this will be fixed I think that a point (1.7.1) release is required.

bhack on Apr 10, 2018