tensorflow: TFLite toco failed to conver quantized model ( mobilenet_v1_1.0_224 ) to tflite format

I could able to convert FaceNet .pb to .tflite model, and following are the instructions to do so:

We will quantise pre-trained Facenet model with 512 embedding size. This model is about 95MB in size before quantization.

$ ls -l model_pc
total 461248
-rw-rw-r--@ 1 milinddeore  staff   95745767 Apr  9  2018 20180402-114759.pb

create a file inference_graph.py with following code:

import tensorflow as tf
from src.models import inception_resnet_v1
import sys
import click
from pathlib import Path

@click.command()
@click.argument('training_checkpoint_dir', type=click.Path(exists=True, file_okay=False, resolve_path=True))
@click.argument('eval_checkpoint_dir', type=click.Path(exists=True, file_okay=False, resolve_path=True))

def main(training_checkpoint_dir, eval_checkpoint_dir):
    traning_checkpoint = Path(training_checkpoint_dir) / "model-20180402-114759.ckpt-275"
    eval_checkpoint = Path(eval_checkpoint_dir) / "imagenet_facenet.ckpt"
    data_input = tf.placeholder(name='input', dtype=tf.float32, shape=[None, 160, 160, 3])
    output, _ = inception_resnet_v1.inference(data_input, keep_probability=0.8, phase_train=False, bottleneck_layer_size=512)
    label_batch= tf.identity(output, name='label_batch')
    embeddings = tf.identity(output, name='embeddings')
    init = tf.global_variables_initializer()
    with tf.Session() as sess:
        sess.run(init)
        saver = tf.train.Saver()
        saver.restore(sess, traning_checkpoint.as_posix())
        save_path = saver.save(sess, eval_checkpoint.as_posix())
        print("Model saved in file: %s" % save_path)

if __name__ == "__main__":
     main()

Run this file on pre-trained model, would generate model for inference. Download pre-trained model and unzip it to model_pre_trained/ directory. Make sure you have python ≥ 3.4 version.

python3 eval_graph.py model_pre_trained/ model_inference/

FaceNet provides freeze_graph.py file, which we will use to freeze the inference model.

python3  src/freeze_graph.py model_inference/  my_facenet.pb

Once the frozen model is generated, time to convert it to .tflite

$ tflite_convert --output_file model_mobile/my_facenet.tflite --graph_def_file my_facenet.pb  --input_arrays "input" --input_shapes "1,160,160,3" --output_arrays embeddings --output_format TFLITE --mean_values 128 --std_dev_values 128 --default_ranges_min 0  --default_ranges_max 6 --inference_type QUANTIZED_UINT8 --inference_input_type QUANTIZED_UINT8

Let us check the quantized model size:

$ ls -l model_mobile/
total 47232
-rw-r--r--@ 1 milinddeore  staff  23667888 Feb 25 13:39 my_facenet.tflite

Interpeter code:

 import numpy as np
 import tensorflow as tf


 # Load TFLite model and allocate tensors.
 interpreter = tf.lite.Interpreter(model_path="/Users/milinddeore/facenet/model_mobile/my_facenet.tflite")
 interpreter.allocate_tensors()

 # Get input and output tensors.
 input_details = interpreter.get_input_details()
 output_details = interpreter.get_output_details()

 # Test model on random input data.
 input_shape = input_details[0]['shape']
 input_data = np.array(np.random.random_sample(input_shape), dtype=np.uint8)
 interpreter.set_tensor(input_details[0]['index'], input_data)

 interpreter.invoke()
 output_data = interpreter.get_tensor(output_details[0]['index'])

 print('INPUTS: ')
 print(input_details)
 print('OUTPUTS: ')
 print(output_details)

Interpeter output:

$ python inout.py
INPUTS:
[{'index': 451, 'shape': array([  1, 160, 160,   3], dtype=int32), 'quantization': (0.0078125, 128L), 'name': 'input', 'dtype': <type 'numpy.uint8'>}]
OUTPUTS:
[{'index': 450, 'shape': array([  1, 512], dtype=int32), 'quantization': (0.0235294122248888, 0L), 'name': 'embeddings', 'dtype': <type 'numpy.uint8'>}]

Hope this helps!

TFLite conversion can be done using SavedModel, i have given the link to the model below. This is as per the documentation here.

I have a model here, which is exported SavedModel using following code:

# SavedModel using simple_save()

ins = {"phase_train_placeholder":phase_train_placeholder}
outs = {"embeddings":embeddings}
tf.saved_model.simple_save(sess, '/content/generated/', ins, outs)

But while converting i am too getting the same error, in this case i am not freezing the model by myself. nor qualitizing it. Below code to convert SavedModel to TFLite:

import tensorflow as tf

saved_model_dir = '/content/generated/'

converter = tf.contrib.lite.TFLiteConverter.from_saved_model(saved_model_dir, input_arrays=['phase_train'], input_shapes=(1,160,160,3), 
                                                             output_arrays=['embeddings'])
tflite_model = converter.convert()
open("converted_model_savedModel.tflite", "wb").write(tflite_model)

Logs:

2018-11-11 02:27:06.150377: I tensorflow/lite/toco/import_tensorflow.cc:1280] Converting unsupported operation: RefSwitch
2018-11-11 02:27:06.150423: I tensorflow/lite/toco/import_tensorflow.cc:1280] Converting unsupported operation: AssignSub
2018-11-11 02:27:06.150476: I tensorflow/lite/toco/import_tensorflow.cc:1280] Converting unsupported operation: RefSwitch
2018-11-11 02:27:06.150510: I tensorflow/lite/toco/import_tensorflow.cc:1280] Converting unsupported operation: AssignSub
2018-11-11 02:27:06.790259: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 5600 operators, 9398 arrays (0 quantized)
2018-11-11 02:27:07.384090: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] After Removing unused ops pass 1: 3582 operators, 6259 arrays (0 quantized)
2018-11-11 02:27:07.836694: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 3582 operators, 6259 arrays (0 quantized)
2018-11-11 02:27:07.839340: F tensorflow/lite/toco/graph_transformations/resolve_batch_normalization.cc:45] Check failed: IsConstantParameterArray(*model, bn_op->inputs[1]) && IsConstantParameterArray(*model, bn_op->inputs[2]) && IsConstantParameterArray(*model, bn_op->inputs[3]) Batch normalization resolution requires that mean, multiplier and offset arrays be constant.
Aborted (core dumped)

You might wanna read the TensorFlow guide on quantization to learn to use fake quantization technique instead of using the transform_graph to do direct quantization.

It seems that link is gone. Add another link here.