opencv: eltwise_layer.cpp:116: error: (-215:Assertion failed) inputs[0] == inputs[i] in function 'getMemoryShapes'

System information (version)

OpenCV => : 4.1.1.26
Operating System / Platform => Ubuntu 18.04
Compiler => ython3.5

Detailed description

Hello I am having the following issue. I am trying to freeze EfficientNet taken from this repo and use the protobuf file for using it with OpenCV dnn module.

The model is a simple classification network. Right after the the Feature Extractor specified in the link, I just try add an extra dense & and an extra classification layer of num_classes.

I have frozen several networks and used them with the dnn library before and I know all the issues that can arise, which I have already tried.

Example code in python (also seen in attached script below):

net = cv2.dnn.readNetFromTensorflow('frozen.pb') 
x_cv = np.random.random((224, 224, 3)).astype(np.uint8)
blob = cv2.dnn.blobFromImage(x_cv, 1.0, (224, 224), (0, 0, 0))
net.setInput(blob)
# (1 x 200)
s = net.forward().shape
print(s)

However life is not that easy. This is the following error:

error: OpenCV(4.1.1) /io/opencv/modules/dnn/src/layers/eltwise_layer.cpp:116: error: (-215:Assertion failed) inputs[0] == inputs[i] in function 'getMemoryShapes'

Eventually I Found out that the problem is caused when using the Squeeze and excitation module (SE module for short). If I disable the flag & remove those modules from the network, the forward pass works.

In my opinion is the flow of the graph of the SE Module that is not acceptable from OpenCV from going to an average pooling to 1x1 convs and finally multiplying the features.

Some remarks:

I had this issue with SE modules in the past, with the difference that the model was using more basic operations such as:

reduce_mean across the features instead of average pooling
fully connected layers instead of 1x1 Convolutions, since they result in the same number of features.

Additionally:

I used the optimize_for_inference_lib tool with no success.
Using the following command: net.getUnconnectedOutLayersNames() gave me the following results:

['block1a_se_squeeze/Mean/flatten', 'block2a_se_squeeze/Mean/flatten', 'block2b_se_squeeze/Mean/flatten', 'block3a_se_squeeze/Mean/flatten', 'block3b_se_squeeze/Mean/flatten', 'block4a_se_squeeze/Mean/flatten', 'block4b_se_squeeze/Mean/flatten', 'block4c_se_squeeze/Mean/flatten', 'block5a_se_squeeze/Mean/flatten', 'block5b_se_squeeze/Mean/flatten', 'block5c_se_squeeze/Mean/flatten', 'block6a_se_squeeze/Mean/flatten', 'block6b_se_squeeze/Mean/flatten', 'block6c_se_squeeze/Mean/flatten', 'block6d_se_squeeze/Mean/flatten', 'block7a_se_squeeze/Mean/flatten', 'avg_pool/Mean/flatten', 'dense_1/Softmax']

Follow up

I would like to know if there is a possible way to add the SE Module either by

reshaping the network in a compatible manner
transforming the graph and using a .pbtxt file additionally
register the module as custom & override the getMemoryShapes()

Of course the easiest solution would be preferrable 😃

Steps to reproduce

Running the attached .py (extension currently txt) file, will follow the pipeline:

Initialize EfficientnetB0 (smallest) with SE modules.
Create a dummy training set and run keras training fit() function.
Save the model as keras extension and reset graph
Load & freeze model with standard keras freezing method. You should see 344 variables frozen in the terminal.
Load model for OpenCV, create dummy blob & run net.forward(). It will crash with the aforementioned error.

opencv_efficientnet.txt

Making the model `freezable`.

If you jump to line 254 of the attached script, you can override the has_se=True to False. This will make the model compatible with OpenCV, proving the detailed description above that this is the factor that makes it show the assertion error.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (7 by maintainers)

Most upvoted comments

@r0l1, As a workaround you may try to apply this patch:

--- a/modules/dnn/src/tensorflow/tf_importer.cpp
+++ b/modules/dnn/src/tensorflow/tf_importer.cpp
@@ -1574,8 +1574,8 @@ void TFImporter::populateNet(Net dstNet)
             }
             else
             {
-                layerParams.set("operation", "prod");
-                int id = dstNet.addLayer(name, "Eltwise", layerParams);
+                layerParams.set("axis", 0);
+                int id = dstNet.addLayer(name, "Scale", layerParams);
                 layer_id[name] = id;
 
                 for (int ii = 0; ii < layer.input_size(); ii++)

Please note that this is a workaround and may break other networks.

dkurt on Dec 19, 2019

Update

By doing some debugging, the layer that causes the problem is this:

x = layers.multiply([x, se_tensor], name=prefix + 'se_excite')

This causes the first SE Module the following:

se_tensor is of shape [1, 1, 1, 32]
x is of shape [1, 112, 112, 32]

Keras allows the element-wise multiplication to happen by using the layers.multiply layer, but not OpenCV. OpenCV complains because the shapes internally are:

x = [1, 32, 112, 112] # Channel first 
se_tensor = [1, 32, 1, 1]

And gives the above error in eltwise_layer.cpp.

Already tried several stuff including:

Reshaping the network to be dense layers.
- This complains that internally does an ExpandDims to keep the flow of the graph, resulting in the following:

cv2.error: OpenCV(4.1.1) /io/opencv/modules/dnn/src/dnn.cpp:525: error: (-2:Unspecified error) Can't create layer "ExpandDims" of type "ExpandDims" in function 'getLayerInstance'

Tiling the se_tensor to become of shape [1, 112, 112, 32] with tf.keras.backend.tile
- This complains that internally now doesn’t have a Tile Layer and gives the same error as above:

cv2.error: OpenCV(4.1.1) /io/opencv/modules/dnn/src/dnn.cpp:525: error: (-2:Unspecified error) Can't create layer "Facenet/lambda/Tile" of type "Tile" in function 'getLayerInstance'

I tried to write the Tile Layer & register it as a custom layer with the following code:

class TileLayer(object):
    def __init__(self, params, blobs):
        self.height = blobs[0][0][1]
        self.width = self.height

    def getMemoryShapes(self, inputs):
        _in, _out = inputs[0], inputs[1]
        return [[_in.shape[0], _in.shape[1], self.height, self.width]]

    def forward(self, inputs):
        return [np.tile(inputs[0], [1, 1, self.height, self.width])]

I am aware the above code might not be correct completely. However I get the following error which I cannot debug:

cv2.error: OpenCV(4.1.1) /io/opencv/modules/dnn/misc/python/pyopencv_dnn.hpp:146: error: (-213:The function/feature is not implemented) Failed to call "getMemoryShapes" method in function 'getMemoryShapes'

Any leads/recommendations ?

harristmv on Sep 20, 2019

@dkurt I just pulled from the latest master branch and included also #16746 for the default backend change. Now everything works if I use the CPU inference engine backend. I’ll test the Myriad target once I am back at work. Thank you for the great work imho. Things are progressing quite fast for OpenCV during the last year.

Edit: Just checked the Myriad target. Works as expected.

r0l1 on Mar 8, 2020