tensorflow: TFLiteConverter adds (de)quantization blocks before and after operations on a weight variable
Hello
I’m trying to convert a TensorFlow model to TFLite. I want the model to maintain a non-trainable state vector that gets updated at every inference. I’ve implemented this using self.add_weight with trainable=False, as shown in the example.
1. System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux KDE Neon 5.26
- TensorFlow installation (pip package or built from source): pip installation
- TensorFlow library (version, if pip package or github SHA, if built from source): 2.11.0
2. Code
import tensorflow as tf
from keras import initializers
# Class Definition
class Net(tf.keras.Model):
def __init__(self):
super(Net, self).__init__()
self.my_weight = self.add_weight("MyWeight",
shape=[1, 10, 1],
initializer=initializers.get('Zeros'),
trainable=False)
@tf.function(input_signature=[tf.TensorSpec(shape=[1, 5, 1], name='MyInput'),])
def call(self, x):
x = x + self.my_weight[:, -5:, :]
self.my_weight.assign(value=tf.concat((self.my_weight[:, -5:, :], x[:, :5, :]), axis=1))
return x
import numpy as np
X = np.random.random((1, 5, 1)).astype(np.float32)
model = Net()
for i in range(5):
Y = model(X, training=False)
# Save Model
model_path = "my_saved_model"
tf.saved_model.save(model, model_path)
model.summary()
def representative_dataset():
for i in range(50):
yield {
"MyInput": np.random.random((1, 5, 1)).astype(np.float32)
}
converter = tf.lite.TFLiteConverter.from_saved_model(model_path)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8 # or tf.uint8
converter.inference_output_type = tf.int8 # or tf.uint8
# Convert the model
tf_lite_model = converter.convert()
# Save the converted model
open('my_tflite_model.tflite', 'wb').write(tf_lite_model)
3. Conversion Issue
The conversion process works fine, but when I draw the model on Netron, it seems that the converter adds quantize/dequantize operations everytime the variable is read or updated. Is there a way to tell the converter to quantize this variable from the start, without having to quantize and dequantize everytime it accesses it?

On another note, my aim is to run such a model on an EdgeTPU, but the EdgeTPU compiler complains about dynamic-sized tensors, without indicating where they are. I believe the culprits are CallOnce, ReadVariable and/or AssignVariable, but I’m not entirely sure how to get around them. Any ideas? Thanks.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 16 (6 by maintainers)
I opened this discussion on the TensorFlow forum. It doesn’t seem to be getting any attention. Do you know if the behavior I’m getting is expected? If so, how can I fix it?
Thanks for the help. I’m not entirely sure I understand what you mean. I don’t want to selectively enable/disable quantization. I want the converter to properly quantize all the weights and biases, including the one I’ve added. From what I understand, TFLiteConverter seems to do this well for other weights (kernel weights in Conv layers, for example), but it strangely doesn’t seem to quantize “MyWeight” and instead needs explicit blocks to do it. Is this supposed to happen? Is it specifically because I’m updating the variable? Also, even if I try to use
tfmot.quantization.keras.quantize_model, I getValueError: `to_quantize` can only either be a tf.keras Sequential or Functional model, which I’m afraid means I can’t quantize the model (using tfmot) as is without rewriting it in one of those two formats. Is that it?Additionally, the workaround you suggested for EdgeTPU gives
ValueError: Cannot set tensor: Got value of type INT8 but expected type FLOAT32 for input 0, name: serving_default_MyInput:0.