tensorflow: ResizeBilinear op mismatch in TF2.0/TFLite

There have been changes in how ResizeBilinear works across the last few versions and its causing issues.

Originally ResizeBilinear op (what you get with tf.image.resize) was completely wrong (see the weird shift below)
This was made less-wrong with the align_corners=True param (defaults to False for compat which makes sense).
None of these match OpenCV, align_corners is not completely terrible but still not right.
for TF2.0 a newer version of the op was made that is really nice and matches OpenCV. PyTorch also matches this version too. The antialias=False (default) param only matters for downscaling.
The tf.keras.layers.UpScaling2D layer calls tf.keras.backend.resize_images
In TF1.x, tf.keras.backend.resize_images called tf.image.resize_bilinear with default args, so got the broken op.
After r1.14 was branched off master, tf.keras.backed.resize_images was changed to use the V2 op.
TF2.0 was branched off master, so has the new good op
TF1.15 was branched off master, so also has the new good op, but is backwards incompatible
TOCO/TFLiteConverter forces any resize_bilinear to use the old broken resize op. In fact it forces align_corners=False without even looking at what the input model has.

TFLite needs to have the new op implemented, and TFLiteConverter needs to distinguish and error if the wrong op is used.

Reproducer:

#!/usr/bin/env python3

import cv2
import numpy as np
import tensorflow as tf
import tensorflow.compat.v1 as tf1
import tensorflow.compat.v2 as tf2

assert tf.version.VERSION == '2.0.0'
np.set_printoptions(precision=2, suppress=True)

start_shape = (2, 2)
resize_shape = (10, 10)
upsample_size = (resize_shape[0] // start_shape[0], resize_shape[1] // start_shape[1])

# A 2x2 image with diagonal corners = 5
a = np.ones((1, start_shape[0], start_shape[1], 1), dtype=np.float32)
a[0, 0, 0, 0] = 5.0
a[0, -1, -1, 0] = 5.0


b = tf.constant(a, dtype=tf.float32)
c = tf1.image.resize(b, resize_shape,
                     method=tf1.image.ResizeMethod.BILINEAR,
                     align_corners=False)
d = tf1.image.resize(b, resize_shape,
                     method=tf1.image.ResizeMethod.BILINEAR,
                     align_corners=True)

e = tf2.image.resize(b, resize_shape,
                     method=tf2.image.ResizeMethod.BILINEAR,
                     antialias=False)
f = tf2.image.resize(b, resize_shape,
                     method=tf2.image.ResizeMethod.BILINEAR,
                     antialias=True)

z = cv2.resize(a[0], resize_shape, interpolation=cv2.INTER_LINEAR)


print("Input: b")
print(b.numpy()[0, :, :, 0])

print("\nTensorflow: c, tf1 align corners False")
print(c.numpy()[0, :, :, 0])

print("\nTensorflow: d, tf1 align corners True")
print(d.numpy()[0, :, :, 0])

print("\nTensorflow: e, tf2 antialias False")
print(e.numpy()[0, :, :, 0])

print("\nTensorflow: f, tf2 antialias True")
print(f.numpy()[0, :, :, 0])

print("OpenCV:")
print(z)


# for TF2.0, the UpSampling2D layer uses tf2 antialias=False
input_layer = tf.keras.layers.Input(shape=(start_shape[0], start_shape[1], 1), batch_size=1)
x = tf.keras.layers.UpSampling2D(size=upsample_size, interpolation='bilinear')(input_layer)
model = tf.keras.models.Model(input_layer, x)

# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

# Load TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()

# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# Test the TensorFlow Lite model on random input data.
input_shape = input_details[0]['shape']
interpreter.set_tensor(input_details[0]['index'], a)

interpreter.invoke()

# The function `get_tensor()` returns a copy of the tensor data.
# Use `tensor()` in order to get a pointer to the tensor.
tflite_results = interpreter.get_tensor(output_details[0]['index'])

# Test the TensorFlow model on random input data.
tf_results = model(b)

print("tflite results shape:", tflite_results.shape, "TF results shape:", tf_results.shape)
print("TFLite results:")
print(tflite_results[0, :, :, 0])

print("\nTF results:")
print(tf_results.numpy()[0, :, :, 0])

print("\ntf1 align_corners=False matches tflite:", np.allclose(c, tflite_results))
print("UpSampling2D: TF matches TFLite:", np.allclose(tf_results, tflite_results))

Output:

Input: b
[[5. 1.]
 [1. 5.]]

Tensorflow: c, tf1 align corners False
[[5.   4.2  3.4  2.6  1.8  1.   1.   1.   1.   1.  ]
 [4.2  3.72 3.24 2.76 2.28 1.8  1.8  1.8  1.8  1.8 ]
 [3.4  3.24 3.08 2.92 2.76 2.6  2.6  2.6  2.6  2.6 ]
 [2.6  2.76 2.92 3.08 3.24 3.4  3.4  3.4  3.4  3.4 ]
 [1.8  2.28 2.76 3.24 3.72 4.2  4.2  4.2  4.2  4.2 ]
 [1.   1.8  2.6  3.4  4.2  5.   5.   5.   5.   5.  ]
 [1.   1.8  2.6  3.4  4.2  5.   5.   5.   5.   5.  ]
 [1.   1.8  2.6  3.4  4.2  5.   5.   5.   5.   5.  ]
 [1.   1.8  2.6  3.4  4.2  5.   5.   5.   5.   5.  ]
 [1.   1.8  2.6  3.4  4.2  5.   5.   5.   5.   5.  ]]

Tensorflow: d, tf1 align corners True
[[5.   4.56 4.11 3.67 3.22 2.78 2.33 1.89 1.44 1.  ]
 [4.56 4.21 3.86 3.52 3.17 2.83 2.48 2.14 1.79 1.44]
 [4.11 3.86 3.62 3.37 3.12 2.88 2.63 2.38 2.14 1.89]
 [3.67 3.52 3.37 3.22 3.07 2.93 2.78 2.63 2.48 2.33]
 [3.22 3.17 3.12 3.07 3.02 2.98 2.93 2.88 2.83 2.78]
 [2.78 2.83 2.88 2.93 2.98 3.02 3.07 3.12 3.17 3.22]
 [2.33 2.48 2.63 2.78 2.93 3.07 3.22 3.37 3.52 3.67]
 [1.89 2.14 2.38 2.63 2.88 3.12 3.37 3.62 3.86 4.11]
 [1.44 1.79 2.14 2.48 2.83 3.17 3.52 3.86 4.21 4.56]
 [1.   1.44 1.89 2.33 2.78 3.22 3.67 4.11 4.56 5.  ]]

Tensorflow: e, tf2 antialias False
[[5.   5.   5.   4.2  3.4  2.6  1.8  1.   1.   1.  ]
 [5.   5.   5.   4.2  3.4  2.6  1.8  1.   1.   1.  ]
 [5.   5.   5.   4.2  3.4  2.6  1.8  1.   1.   1.  ]
 [4.2  4.2  4.2  3.72 3.24 2.76 2.28 1.8  1.8  1.8 ]
 [3.4  3.4  3.4  3.24 3.08 2.92 2.76 2.6  2.6  2.6 ]
 [2.6  2.6  2.6  2.76 2.92 3.08 3.24 3.4  3.4  3.4 ]
 [1.8  1.8  1.8  2.28 2.76 3.24 3.72 4.2  4.2  4.2 ]
 [1.   1.   1.   1.8  2.6  3.4  4.2  5.   5.   5.  ]
 [1.   1.   1.   1.8  2.6  3.4  4.2  5.   5.   5.  ]
 [1.   1.   1.   1.8  2.6  3.4  4.2  5.   5.   5.  ]]

Tensorflow: f, tf2 antialias True
[[5.   5.   5.   4.2  3.4  2.6  1.8  1.   1.   1.  ]
 [5.   5.   5.   4.2  3.4  2.6  1.8  1.   1.   1.  ]
 [5.   5.   5.   4.2  3.4  2.6  1.8  1.   1.   1.  ]
 [4.2  4.2  4.2  3.72 3.24 2.76 2.28 1.8  1.8  1.8 ]
 [3.4  3.4  3.4  3.24 3.08 2.92 2.76 2.6  2.6  2.6 ]
 [2.6  2.6  2.6  2.76 2.92 3.08 3.24 3.4  3.4  3.4 ]
 [1.8  1.8  1.8  2.28 2.76 3.24 3.72 4.2  4.2  4.2 ]
 [1.   1.   1.   1.8  2.6  3.4  4.2  5.   5.   5.  ]
 [1.   1.   1.   1.8  2.6  3.4  4.2  5.   5.   5.  ]
 [1.   1.   1.   1.8  2.6  3.4  4.2  5.   5.   5.  ]]
OpenCV:
[[5.   5.   5.   4.2  3.4  2.6  1.8  1.   1.   1.  ]
 [5.   5.   5.   4.2  3.4  2.6  1.8  1.   1.   1.  ]
 [5.   5.   5.   4.2  3.4  2.6  1.8  1.   1.   1.  ]
 [4.2  4.2  4.2  3.72 3.24 2.76 2.28 1.8  1.8  1.8 ]
 [3.4  3.4  3.4  3.24 3.08 2.92 2.76 2.6  2.6  2.6 ]
 [2.6  2.6  2.6  2.76 2.92 3.08 3.24 3.4  3.4  3.4 ]
 [1.8  1.8  1.8  2.28 2.76 3.24 3.72 4.2  4.2  4.2 ]
 [1.   1.   1.   1.8  2.6  3.4  4.2  5.   5.   5.  ]
 [1.   1.   1.   1.8  2.6  3.4  4.2  5.   5.   5.  ]
 [1.   1.   1.   1.8  2.6  3.4  4.2  5.   5.   5.  ]]
tflite results shape: (1, 10, 10, 1) TF results shape: (1, 10, 10, 1)
TFLite results:
[[5.   4.2  3.4  2.6  1.8  1.   1.   1.   1.   1.  ]
 [4.2  3.72 3.24 2.76 2.28 1.8  1.8  1.8  1.8  1.8 ]
 [3.4  3.24 3.08 2.92 2.76 2.6  2.6  2.6  2.6  2.6 ]
 [2.6  2.76 2.92 3.08 3.24 3.4  3.4  3.4  3.4  3.4 ]
 [1.8  2.28 2.76 3.24 3.72 4.2  4.2  4.2  4.2  4.2 ]
 [1.   1.8  2.6  3.4  4.2  5.   5.   5.   5.   5.  ]
 [1.   1.8  2.6  3.4  4.2  5.   5.   5.   5.   5.  ]
 [1.   1.8  2.6  3.4  4.2  5.   5.   5.   5.   5.  ]
 [1.   1.8  2.6  3.4  4.2  5.   5.   5.   5.   5.  ]
 [1.   1.8  2.6  3.4  4.2  5.   5.   5.   5.   5.  ]]

TF results:
[[5.   5.   5.   4.2  3.4  2.6  1.8  1.   1.   1.  ]
 [5.   5.   5.   4.2  3.4  2.6  1.8  1.   1.   1.  ]
 [5.   5.   5.   4.2  3.4  2.6  1.8  1.   1.   1.  ]
 [4.2  4.2  4.2  3.72 3.24 2.76 2.28 1.8  1.8  1.8 ]
 [3.4  3.4  3.4  3.24 3.08 2.92 2.76 2.6  2.6  2.6 ]
 [2.6  2.6  2.6  2.76 2.92 3.08 3.24 3.4  3.4  3.4 ]
 [1.8  1.8  1.8  2.28 2.76 3.24 3.72 4.2  4.2  4.2 ]
 [1.   1.   1.   1.8  2.6  3.4  4.2  5.   5.   5.  ]
 [1.   1.   1.   1.8  2.6  3.4  4.2  5.   5.   5.  ]
 [1.   1.   1.   1.8  2.6  3.4  4.2  5.   5.   5.  ]]

tf1 align_corners=False matches tflite: True
UpSampling2D: TF matches TFLite: False

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 22 (21 by maintainers)

Commits related to this issue

TFLite export with builtin-ops for TF-2.2 The resize-bilinear op was missing support for half-pixel-centers: https://github.com/tensorflow/tensorflow/issues/33691 TFLite export with only the builtin... — committed to perfinion/yolov3-tf2 by perfinion 4 years ago
TFLite export with builtin-ops for TF-2.2 The resize-bilinear op was missing support for half-pixel-centers: https://github.com/tensorflow/tensorflow/issues/33691 TFLite export with only the builtin... — committed to leigh-johnson/yolov3-tf2 by perfinion 4 years ago

Most upvoted comments

I have implemented the new version, its currently under review. This op and its code is used at multiple places in internal tests, so I had to ensure nothing was broken 😃. This should go in within a week or so. Sorry about the delay!

srjoglekar246 on Jan 14, 2020

This change has gone in to TensorFlow’s master branch. @hgaiser May you confirm and close this issue if it works?

srjoglekar246 on Jan 24, 2020

The latter 😃. Since it affects a bunch of stuff, the review is taking time. I will ping back on this issue as soon as its in. Thanks for the patience!

srjoglekar246 on Jan 22, 2020

We haven’t pushed any changes yet, but this should be fixed by end of the year. Sorry for the delay!

srjoglekar246 on Nov 27, 2019