model-optimization: Failed with Post Training Quantization after Quantization Aware Training

Also, regarding HardSwish if you have the time and are interested, I’m happy to guide you in how to implement support for it 😃

Regarding MobileNetV2 reproduction, looking at your code it seems you are training on CIFAR. It won’t be as straight-forward to reproduce the full training.

We trained a Keras MobileNet V2 model with hyperparams from this. We then quantized the model and trained again for a few epochs.

I think the reason your conversion code is failing is due to

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]

Try removing it, and I think conversion should work. If it doesn’t, please let me know. Basically, the QAT conversion by default uses Float inputs/outputs based on the model signature. There is work in progress in TFLiteConverterV2 to support a different model interface int8/uint8 etc.

See this.

Hope this helps.

Hi @kalaluthien,

Thanks for the well thought out and detailed bug report. Sorry for the delay in getting back - there’s generally limited time I take out each week to look at github issues 😃

1. AveragePooling2D
2. MaxPooling2D

I tried out both these example and they converted+ran just fine for me. Perhaps, there is a flag you are missing during conversion. Check this file for conversion code.

Looking at your colab code, converter._experimental_new_quantizer = True is missing. Please try that and let me know how it goes.

1. TensorflowOpLayer

This failure is expected. By default, our goal is to support built-in keras layers which is basically layers under the tf.keras.layers module. TensorflowOpLayer can be used to wrap any TF op, and it’s not feasible to meaningfully cover any tf op.

The recommended approach here is to use built-in Keras layers to achieve this. So you can use tf.keras.layers.Add and tf.keras.layers.Reshape instead of using + and expand_dims. That should solve it.

If you really do want to use something else, it’s the user’s responsibility to provide an appropriate QuantizeConfig for your use.

1. HardSwish

This is again the same problem as TensorFlowOpLayer. And yes, you are right the existing pattern only matches Conv+BN+ReLU. The code likely became slow since it had added a bunch of Quant/Dequant ops in between. I don’t think the converter is likely to match Conv/BN/(Add+Mul ops matching hardswish) either while folding.

The proper fix here would be to add support for hardswish. Can you please file a separate bug for that requesting HardSwish support. I’ll take some time out to add it. We covered MobileNet v1/v2, so this is currently missing.

But we should be able to add support for this. We also need to ensure the converter is handling it properly.

1. GlobalAveragePooling-Dense

Again, this works for me. That’s how we got MobileNetV2 working and created the results. Perhaps, this is the same issue as Averge/MaxPooling. Please try the fix out and let me know if that works.

Seems like this bug is solved. I’m closing it, please feel free to reopen.

You can start a new issue for the hardswish and we can continue our conversation there. Even if it’s done in your code, it can remain an example for others to follow.

And it’ll be pretty easy to incorporate into the library once you’ve implemented it. We can try and get HardSwish moved into Keras.

Thanks @kalaluthien for your patience and proactive use of the library.

As for HardSwish, I just looked into it a bit. There seem to be a few tricky pieces.

For starters, hard_swish has not been added as an activation in Keras yet. The goal of the tfmot library is to provide default behavior for all built-in Keras layers/activations. But since hard_swish is not a built-in activation yet, we can’t really add a pattern matching it in the library code. It would need to be handled by the user.

I would recommend adding support for it in your code to begin with. Once hard_swish gets added, we can move this code internally. You should be able to file a bug on keras/tf to check whether they plan to add support for it.

You can create a class HardSwish(Layer) which gets added after Conv + BN. You should be able to use built-in Add and Multiply to do so.

Next, to understand exactly what support needs to be added, we would need to understand how it executes in TFLite.

I created a simple model.

inp = tf.keras.Input(shape=(28, 28, 1))
x = tf.keras.layers.Conv2D(32, 5)(inp)
x = tf.keras.layers.BatchNormalization()(x)
x = tf.keras.layer.Relu(6.0)(x + 3.0) * 1.67
m = tf.keras.Model(inp, x)

m.save('hswish.h5')

Converted it using the following code.

conv = tf.lite.TFLiteConverter.from_keras_model(m)
def representative_dataset_gen():
  for _ in range(num_calibration_steps):
     yield np.random.rand(28, 28, 1)

conv.representative_dataset = representative_dataset_gen
conv.convert()

# saved as hswish.tflite

As you can see the converter fuses the Add into the bias, but the Mul comes after.

So you’ll need to place the FakeQuant after the Add + ReLU but before the Mul. And likely use a transform similar to this. That should sort the issue out.

Oh I’m sorry I made a mistake. I meant use

converter.experimental_new_converter = True

That’s what was missing.