tensorflow: Bug in Transpose Convolution Padding
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
- TensorFlow installed from (source or binary): Binary
- TensorFlow version (use command below): 2.3.0
- Python version: 3.7.6
- Bazel version (if compiling from source): N/A
- GCC/Compiler version (if compiling from source): N/A
- CUDA/cuDNN version: 10.2
- GPU model and memory: Nvidia GeForce GTX 1080 8GB
Describe the current behavior Padding for transpose convolutions under certain parameter combinations fails to distribute the padding as expected. This can, in extreme cases, lead to highly skewed outputs. This can happen regardless of whether the filter or signal are even or odd.
Here, the output is subtly different but quite simple to verify that it must have been generated with a padding of [1, 0] rather than [0, 1]:
TESTING 2 x 2 INPUT WITH 2 x 2 KERNEL, (1, 1) STRIDE AND "SAME" PADDING:
Input:
[[1 1]
[1 1]]
Kernel:
[[1 1]
[1 1]]
Dilated Input: (=> 2 x 2 array)
[[1 1]
[1 1]]
Dilated Input + Padding: (=> 3 x 3 array)
[[1 1 0]
[1 1 0]
[0 0 0]]
Manual Result: (=> 2 x 2 array)
[[4 2]
[2 1]]
Standard Result: (=> 2 x 2 array)
[[1 2]
[2 4]]
Here’s an example with an odd kernel and valid padding instead:
TESTING 2 x 2 INPUT WITH 3 x 3 KERNEL, (5, 5) STRIDE AND "VALID" PADDING:
Input:
[[1 1]
[1 1]]
Kernel:
[[1 1 1]
[1 1 1]
[1 1 1]]
Dilated Input: (=> 6 x 6 array)
[[1 0 0 0 0 1]
[0 0 0 0 0 0]
[0 0 0 0 0 0]
[0 0 0 0 0 0]
[0 0 0 0 0 0]
[1 0 0 0 0 1]]
Dilated Input + Padding: (=> 12 x 12 array)
[[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 1 0 0 0 0 1 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 1 0 0 0 0 1 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]]
Manual Result: (=> 10 x 10 array)
[[0 0 0 0 0 0 0 0 0 0]
[0 1 1 1 0 0 1 1 1 0]
[0 1 1 1 0 0 1 1 1 0]
[0 1 1 1 0 0 1 1 1 0]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0]
[0 1 1 1 0 0 1 1 1 0]
[0 1 1 1 0 0 1 1 1 0]
[0 1 1 1 0 0 1 1 1 0]
[0 0 0 0 0 0 0 0 0 0]]
Standard Result: (=> 10 x 10 array)
[[1 1 1 0 0 1 1 1 0 0]
[1 1 1 0 0 1 1 1 0 0]
[1 1 1 0 0 1 1 1 0 0]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0]
[1 1 1 0 0 1 1 1 0 0]
[1 1 1 0 0 1 1 1 0 0]
[1 1 1 0 0 1 1 1 0 0]
[0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0]]
Here’s an example with an odd signal and even kernel:
TESTING 3 x 3 INPUT WITH 2 x 2 KERNEL, (4, 4) STRIDE AND "VALID" PADDING:
Input:
[[1 1 1]
[1 1 1]
[1 1 1]]
Kernel:
[[1 1]
[1 1]]
Dilated Input: (=> 9 x 9 array)
[[1 0 0 0 1 0 0 0 1]
[0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0]
[1 0 0 0 1 0 0 0 1]
[0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0]
[1 0 0 0 1 0 0 0 1]]
Dilated Input + Padding: (=> 13 x 13 array)
[[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 1 0 0 0 1 0 0 0 1 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 1 0 0 0 1 0 0 0 1 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 1 0 0 0 1 0 0 0 1 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]]
Manual Result: (=> 12 x 12 array)
[[0 0 0 0 0 0 0 0 0 0 0 0]
[0 1 1 0 0 1 1 0 0 1 1 0]
[0 1 1 0 0 1 1 0 0 1 1 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 1 1 0 0 1 1 0 0 1 1 0]
[0 1 1 0 0 1 1 0 0 1 1 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 1 1 0 0 1 1 0 0 1 1 0]
[0 1 1 0 0 1 1 0 0 1 1 0]
[0 0 0 0 0 0 0 0 0 0 0 0]]
Standard Result: (=> 12 x 12 array)
[[1 1 0 0 1 1 0 0 1 1 0 0]
[1 1 0 0 1 1 0 0 1 1 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[1 1 0 0 1 1 0 0 1 1 0 0]
[1 1 0 0 1 1 0 0 1 1 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[1 1 0 0 1 1 0 0 1 1 0 0]
[1 1 0 0 1 1 0 0 1 1 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]]
Finally, here the skew is quite large and noticeable due to the large amount of padding, which clearly seems like it could only have stemmed from the padding being allocated entirely to the bottom and right only:
TESTING 2 x 2 INPUT WITH 2 x 2 KERNEL, (6, 6) STRIDE AND "SAME" PADDING:
Input:
[[1 1]
[1 1]]
Kernel:
[[1 1]
[1 1]]
Dilated Input: (=> 7 x 7 array)
[[1 0 0 0 0 0 1]
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]
[0 0 0 0 0 0 0]
[1 0 0 0 0 0 1]]
Dilated Input + Padding: (=> 13 x 13 array)
[[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 1 0 0 0 0 0 1 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 1 0 0 0 0 0 1 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0]]
Manual Result: (=> 12 x 12 array)
[[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 1 1 0 0 0 0 1 1 0 0]
[0 0 1 1 0 0 0 0 1 1 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 1 1 0 0 0 0 1 1 0 0]
[0 0 1 1 0 0 0 0 1 1 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]]
Standard Result: (=> 12 x 12 array)
[[1 1 0 0 0 0 1 1 0 0 0 0]
[1 1 0 0 0 0 1 1 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[1 1 0 0 0 0 1 1 0 0 0 0]
[1 1 0 0 0 0 1 1 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]]
Describe the expected behavior Padding for transpose convolutions should be distributed evenly with at most an extra zero on the right when not evenly divisible by 2. This is TensorFlow convention for regular convolutions, but not here, oddly enough.
All of my examples are easy to compute and verify by hand. It is not clear what the exact method of allocating the padding is for transpose convolutions, as the documentation is quite scarce. Just to clarify, my code does actually match up for many parameter combinations I did not show here for brevity, but it disagrees with enough to make me suspicious that something’s up. I would love it if someone could explain the algorithm better!
Standalone code to reproduce the issue I have provided code to replicate the above examples below:
# Import modules
import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Conv2D, Conv2DTranspose
# Define convenience functions
def dilate_array(x, kernel, dilation_rate, padding = 'valid'): # manually dilate array by inserting zeros
# Get padding for after dilation
rank = len(x)
x = x[None, ..., None]
paddings, _ = get_deconv_padding(x, kernel, strides, padding = padding) # calculate BEFORE dilation
# Create "upsampled" array via repetition then cut off extra values by slicing
shp, strd, slc = (1, ), (slice(None, None, None), ), (slice(None, None, None), )
dil = inp = Input(x.shape[1:])
for i in range(rank):
shp += (dilation_rate[i]*dil.shape[i + 1] - dilation_rate[i] + 1, )
dil = tf.repeat(dil, dilation_rate[i], axis = i + 1)
strd += (slice(None, None, dilation_rate[i]), )
slc += (slice(None, shp[i + 1], None), )
shp += (1, )
strd += (slice(None, None, None), )
slc += (slice(None, None, None), )
dil = dil[slc]
# Create mask to replace repeats with zeros a la "proper" resampling and transpose convolution
mask = np.zeros(shp)
mask[strd] = 1.0
mask = tf.cast(mask, dil.dtype)
# Mask dilated input
dil = mask*dil
# Pad dilated input
pad = tf.pad(dil, paddings, mode = 'CONSTANT')
# Execute dilation + padding
model = Model(inp, [dil, pad])
xd, xp = model.predict(x)
xd, xp = xd[0, ..., 0], xp[0, ..., 0]
return xd, xp, paddings
def deconv_padding_after_dilation(input_length, kernel_length, stride, padding = 'valid'): # adjusted padding for manual dilation (== stride)
# Compute minimum padding length assuming manually dilated array
output_length = stride*input_length + (max(kernel_length - stride, 0) if padding == 'valid' else 0)
dilated_length = stride*input_length - (stride - 1)
pad_length = output_length - dilated_length + kernel_length - 1
return pad_length, output_length
def get_deconv_padding(x, kernel, strides, padding = 'valid'):
# Calculate needed paddings with PRE-DILATED input to apply to DILATED input
rank = len(x.shape) - 2
paddings, output_shape = [[0, 0]], (1, )
for i in range(rank):
pad_length, output_length = deconv_padding_after_dilation(int(x.shape[i + 1]), kernel.shape[i], strides[i], padding = padding)
paddings += [[pad_length//2, pad_length//2 + pad_length%2]]
output_shape += (output_length, )
paddings += [[0, 0]]
output_shape += (1, )
return paddings, output_shape
def execute_manual_transpose_convolution(x, kernel, strides, padding = 'valid'):
# Manually dilate and pad output then convolve normally
xd, xp, paddings = dilate_array(x, kernel, strides, padding = padding)
inp = Input(xp.shape + (1, ))
conv = Conv2D(1, kernel.shape, padding = 'valid')(inp)
model = Model(inp, conv)
model.set_weights([kernel[..., None, None], np.zeros((1, ))])
y = model.predict(xp[None, ..., None])[0, ..., 0]
return xd, xp, paddings, y
def execute_standard_transpose_convolution(x, kernel, strides, padding = 'valid'):
# Use standard transpose convolution
inp = Input(x.shape + (1, ))
convT = Conv2DTranspose(1, kernel.shape, strides = strides, padding = padding)(inp)
model = Model(inp, convT)
model.set_weights([kernel[..., None, None], np.zeros((1, ))])
y = model.predict(x[None, ..., None])[0, ..., 0]
return y
# Define inputs
rank = 2
x = np.ones(rank*(2, ), dtype = 'float32') # CHANGE THIS
kernel = np.ones(rank*(2, ), dtype = 'float32') # CHANGE THIS
strides = rank*(1, ) # CHANGE THIS
padding = 'same' # CHANGE THIS
# Test transpose convolution
xd, xp, paddings, y_manual = execute_manual_transpose_convolution(x, kernel, strides, padding = padding)
y_standard = execute_standard_transpose_convolution(x, kernel, strides, padding = padding)
# Show results
print('TESTING {} INPUT WITH {} KERNEL, {} STRIDE AND "{}" PADDING:'.format(' x '.join([str(s) for s in x.shape]),
' x '.join([str(s) for s in kernel.shape]),
strides,
padding.upper()))
print('Input:')
print(x.astype(int))
print('')
print('Kernel:')
print(kernel.astype(int))
print('\n')
print('Dilated Input: (=> {} array)'.format(' x '.join([str(s) for s in xd.shape])))
print(xd.astype(int))
print('')
print('Dilated Input + Padding: (=> {} array)'.format(' x '.join([str(s) for s in xp.shape])))
print(xp.astype(int))
print('\n')
print('Manual Result: (=> {} array)'.format(' x '.join([str(s) for s in y_manual.shape])))
print(y_manual.astype(int))
print('')
print('Standard Result: (=> {} array)'.format(' x '.join([str(s) for s in y_standard.shape])))
print(y_standard.astype(int))
print('')
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 16 (6 by maintainers)
@carrycooldude , Any updates on this issue? Please let us know if you are still working on this issue