edgetpu: Compiler fails to compile int8 quantized TFLite Model

Description

I want to convert a int8 quantized tflite model:

converter.optimizations = [tf.lite.Optimize.DEFAULT]
def representative_dataset_gen():
  # Get sample input data as a numpy array in a method of your choosing.
  for data in dataset:
    # not sure why, for single input data use [data] for multiple inputs just data...
    yield [data]
converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
converter.experimental_new_converter = True
tflite_model = converter.convert()

I can load the quantized tflite model without issues and run inference with it. But when I want to convert it the edgetpu compiler fails with many of these errors:

Edge TPU Compiler version 16.0.384591198
Searching for valid delegate with step 1
Try to compile segment with 136 ops
Started a compilation timeout timer of 180 seconds.
ERROR: :344 no_integer_overflow_from_quantization was not true.
ERROR: Node number 31 (CONV_2D) failed to prepare.

Compilation failed: Model failed in Tflite interpreter. Please ensure model can be loaded/run in Tflite interpreter.
Compilation child process completed within timeout period.
Compilation failed!

I assume this means that somewhere in the quantized model there are integers outside of int8 range. But that should not happen with the quantized tflite model, right? model_quant.tflite.zip

And thanks for adding more meaningful error messages with the new compiler update!

Click to expand!

Issue Type

Bug, Support

Operating System

Linux

Coral Device

No response

Other Devices

No response

Programming Language

Python 3.7

Relevant Log Output

No response

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 17 (7 by maintainers)

Most upvoted comments

I have a similar problem with compilation. Not sure if it should be part of this thread or a separate thread, but it seems to be an issue with weight conversion to int8

My model compiles when I create it as long as I don’t load my Tensorflow weights:

If I create the model, don’t train or load weights, quantize and compile with the same settings as in this thread, things work.
When I load pre-trained weights, and repeat the procedure (quantize and compile), it doesn’t work and fails without an error message: edgetpu_compiler -a -s -t 1000 {save_path} returns:

Edge TPU Compiler version 16.0.384591198
Started a compilation timeout timer of 1000 seconds.
Compilation child process completed within timeout period
Compilation failed!

When pre-trained weights are not loaded, the log file shows the following layers:

Edge TPU Compiler version 16.0.384591198  
Input: /home/ubuntu/network_quant.tflite   
Output: network_quant_edgetpu.tflite  
  
Operator                       Count      Status
CONCATENATION                  3          Mapped to Edge TPU
SOFTMAX                        1          Mapped to Edge TPU
RESHAPE                        2          Mapped to Edge TPU
ADD                            19         Mapped to Edge TPU
QUANTIZE                       2          Mapped to Edge TPU
TRANSPOSE_CONV                 3          Mapped to Edge TPU
CONV_2D                        59         Mapped to Edge TPU

Software versions: Tensorflow 2.8.0, Python 3.9.12

converter.optimizations = [tf.lite.Optimize.DEFAULT] 
converter.representative_dataset = representative_data_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8

estebanvalles on Jul 1, 2022

Hi, @j-o-d-o can you please try commenting out this line:

converter.experimental_new_converter = True

if problem still persist, can you please share your saved model?

hjonnala on Aug 26, 2021