tensorflow: Keras: removing layers with model.layers.pop() doesn't work

I got a solution by using model._layers.pop(0) instead of model.layers.pop(0)

@nairouz There are two issues in your code:

• You can’t use model.layers.pop() to remove the last layer in the model. In tf.keras, model.layers will return a shallow copy version of the layers list, so actually you don’t remove that layer, just remove the layer in the return value.
• If you want to remove the last dense layer and add your own one, you should use hidden = Dense(120, activation='relu')(model.layers[-2].output). model.layers[-1].output means the last layer’s output which is the final output, so in your code, you actually didn’t remove any layers.

Thanks.

The purpose of model.layers.pop() is to remove the last layer and fine-tune it reserving the already-trained weights. I resolved the problem simply as below.

vgg16_model = keras.applications.vgg16.VGG16() model = Sequential() for layer in vgg16_model.layers[:-1]: # just exclude last layer from copying model.add(layer) for layer in model.layers:
layer.trainable = False model.add(Dense(2, activation=‘softmax’))

With model.summary(), you can confirm that the last layer of the model has been replaced.

Faced the same issue. Try popping the last layer after loading the model vgg16_model = keras.applications.vgg16.VGG16() vgg16_model.layers.pop() Then, if you want to create a Sequential one just iterrate over it model = Sequential() for layer in vgg16_model.layers: model.add(layer) and then add the other layer(s) that you want model.add(Dense(3, activation=‘softmax’))

That should work

Only thing which worked in my case (with TF 2.0.0) was :

model = tf.keras.models.load_model(path_model)
model.pop()  # to remove last layer
new_model = tf.keras.Sequential([
	model,
	tf.keras.layers.Dense(units=nb_classes, activation="softmax", name="new_layer_name")
])
new_model.build((None, height, width, 3))

Here is an example of how to remove the last layer of a pretrained model in tf.keras (NOT keras!) and add a new output layer. This method could be used to change the number of classes for a classification problem, or to change the model to for a regression problem. The example below changes a 10 class classification model to a 2 output regression model using InceptionV3 in tf.keras.

Create InceptionV3 model, load pre-trained weights, find layer you want add on to

nb_classes = 10; # number must match your pre-trained weights base_model =InceptionV3(include_top=True, weights=weightFile, input_shape=input_shape, classes=nb_classes) print(base_model.summary()) # find layer in the summary that you want to add new output to

base_output = base_model.layers[311].output # layer number obtained from model summary above

attach new desired output layer to the base model to create your new model

num_regression_targets = 2 # some point (x, y) new_output = tf.keras.layers.Dense(activation=“sigmoid”, units=num_regression_targets)(base_output) new_model = tf.keras.models.Model(inputs=base_model.inputs, outputs=new_output) print(new_model.summary())

Now the new_model is ready to use

@yanboliang Thank you very much for the response. So basically, there is no need to use model.layers.pop() in this case if I understood the point. Actually, I think the pop method implementation has changed since I have found many examples where it is used to remove a layer from a model. You can check:

1. https://stackoverflow.com/questions/41668813/how-to-add-and-remove-new-layers-in-keras-after-loading-weights
2. https://github.com/keras-team/keras/issues/8909

Can someone please provide an example of the best way to do this for tf.keras? All I want to do is remove the last layer of a pretrained model

I re-rad my post above, and realize the explanation could be clearer. Basically, the strategy is to create a new model starting from a base model that is a standard model included in tf.keras for which you have pre-trained weights. You can find the standard models are included in tf.keras.keras.applications (tensorflow 1.13) here:

https://github.com/keras-team/keras-applications

You create an instance of the base model, then look for the layer which feeds into the final fully connected layer (or other desired layer) by looking at the model summary. Then you create a new layer which takes as input the desired layer output from the base model. Finally, you define a new model in tf.keras.model.Models() by defining the inputs and outputs. Tensorflow will now have a graph of the base model with the new output layer.

pop doesn’t help anything here, the correct way to do it is:

model2 = Model(model.input, model.layers[-2].output)

Tested this only for a non-sequential model.

@ironmanciti @AthanasiosTsiaras This solution gives me

TypeError: The added layer must be an instance of class Layer. Found: <tensorflow.python.keras.engine.input_layer.InputLayer object at 0x7f9869a25cc0>

Previous replies mention iterating over base model and adding to a new model, which seems ok for VGG16. For MobileNetV2 it seems to not be possible

when iterating over its layers it will raise an error when reaching an merge.Add layer <tensorflow.python.keras.layers.merge.Add object at 0x000002BDB32D0F48>

ValueError: A merge layer should be called on a list of inputs.

I understood from previous conversation that using model.layers will bring a shallow copy from the model and using model._layers would bring the actual layers. I observed that as @cgebbe mentioned will indeed remove the layers (at least from summary view)

I got a solution by using model._layers.pop(0) instead of model.layers.pop(0)

I didn’t found any cons of using this method, but usually messing around the private vars is not a safe thing to do if you don’t know the details of the implementation. Does anybody found any problem with this method ? Another question would be how I can properly add the mege.Add object layer to my model if I would try the iteration method.

Thanks!

I think that’s because the structure of mobilenetV2 isn’t “sequential”. For the method that you iterate through all the layers, you assume the model structure passes data through layer by layer. This method is available for VGG since VGG is this kind of model. Yet some model structures aren’t like this, such as resnet which contains shortcuts for some data. In these kind of cases, including mobilenetV2, you have to use the functional api solution provided by @cmwolverton here.

Previous replies mention iterating over base model and adding to a new model, which seems ok for VGG16. For MobileNetV2 it seems to not be possible

when iterating over its layers it will raise an error when reaching an merge.Add layer <tensorflow.python.keras.layers.merge.Add object at 0x000002BDB32D0F48>

ValueError: A merge layer should be called on a list of inputs.

I understood from previous conversation that using model.layers will bring a shallow copy from the model and using model._layers would bring the actual layers. I observed that as @cgebbe mentioned will indeed remove the layers (at least from summary view)

I got a solution by using model._layers.pop(0) instead of model.layers.pop(0)

I didn’t found any cons of using this method, but usually messing around the private vars is not a safe thing to do if you don’t know the details of the implementation. Does anybody found any problem with this method ? Another question would be how I can properly add the mege.Add object layer to my model if I would try the iteration method.

Thanks!

for me following code is a nice workaround. It builds another model without the last layer. Thus effectively being .pop()

model_reduced = Sequential()

for layer in old_model.layers[:-1]:
  model_reduced.add(layer)