kornia: [Bug] warp_perspective does not return the input if used with identity matrix

🐛 Bug

The output of kornia.geometry.warp_perspective does not equal the input if the identity matrix is used.

To Reproduce

Steps to reproduce the behavior:

import torch
from kornia.geometry import warp_perspective

torch.manual_seed(0)

dsize = (32, 16)
src = torch.rand(1, 3, *dsize)
M = torch.eye(3).unsqueeze(0)

dst = warp_perspective(src, M, dsize)

mae = torch.mean(torch.abs(dst - src))
print(mae.item())

0.14952071011066437

Expected behavior

0.0

Environment

kornia==0.4.1

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 20 (15 by maintainers)

Most upvoted comments

I agree with the deprecation. It is further supported by the fact that no other library seems to have align_corners or equivalent for image-transform-by-matrix functions:

In my opinion the addition of an align_corners option is confusing, of limited practical use, and increases the likelihood of doing bad things like transforming images and point annotations in subtly different ways despite using the same transformation matrix.

anibali on Oct 7, 2021

I agree with @pmeier that we might add it in the docs to let the users more aware of the differences of the parameter choice. This pic is a nice demonstration.

shijianjian on Mar 11, 2021

@Multihuntr assuming that possibly makes sense to get rid of align_corners from warp_affine, warp_perspective. This might requires some work to verify that we don’t break any other component since we highly rely on this operator.

@pmeier @anibali @ducha-aiki any opinions about this deprecation ? I think that we introduced support to align_corners because pytorch did, but we never discussed the real need for us.

edgarriba on Oct 7, 2021

Our initial intention here was to mimic opencv behaviour. In the early stages, pytorch had default align_corners==True which as showed here is what we mostly support and assume by default. Was later that align_córners was introduced to generate the grid and overcomplicated our logic. Possibly the question is that I from Kornia should be really expose align_corners at all and if it makes sense for our users. /cc @ducha-aiki

This is possibly one of the most core functionalities in the lib so I’d love to hear opinions and use cases on that end.

edgarriba on Oct 4, 2021

I was investigating how the different libraries handle the sampling issues of interpolating, and I wrote a test script that compared PyTorch, OpenCV and Kornia on a few different settings. There seems to be a few inconsistencies between Kornia and the others. I apologise but I don’t have the solution to it. It may be an issue in the PyTorch functions you call?

With identity matrix:
- source image == cv2.warp_affine == cv2.warp_perspective == cv2.resize == torch.nn.Upsample (regardless of align_corners)
- With align_corners=False:
  - kornia.geometry.warp_affine == source ✔️
  - kornia.geometry.warp_perspective cuts off bottom right and uses border constant of 0. ❗
- With align_corners=True:
  - kornia.geometry.warp_affine testing a 3x3 stretches top-left 2x2 into the space of 3x3. I suspect an off-by-one in indexing. ❗
  - kornia.geometry.warp_perspective == source ✔️
With affine transformation scaling by factor of 2:
- cv2.warp_affine == cv2.warp_perspective
- With align_corners=False:
  - cv2.resize == torch.nn.Upsample which is not quite the same as the diagram with blue dots implies;
  - kornia.geometry.warp_affine != kornia.geometry.warp_perspective != cv2.warp_affine:
    - kornia.geometry.warp_affine cuts off bottom right and uses border constant of 0 ❗
    - kornia.geometry.warp_perspective cuts off bottom right, ends up with a border of 0s ❗
- With align_corners=True:
  - torch.nn.Upsample as per the diagram with the blue dots.
  - kornia.geometry.warp_affine == kornia.geometry.warp_perspective == cv2.warpAffine == cv2.warpPerspective ✔️
    - What’s the justification for calling this “align_corners=True”? It’s not using a fixed spacing between sampled points like in torch.nn.Upsample.

Here's my code (click to reveal)

import torch
import torch.nn
import torch.nn.functional as F

import kornia.geometry
import cv2
import numpy as np

print('Kornia version', kornia.__version__)

src = torch.tensor([
    [ 0,  2,  4,],
    [ 2,  4,  6,],
    [ 4,  6,  8,],
], dtype=torch.float32)
src_th = src[None, None]
src_cv2 = src.numpy()[..., None]
size = np.array(src.shape)

M1 = torch.eye(3).unsqueeze(0)
S = 2
M2 = torch.tensor([
    [S, 0, 0],
    [0, S, 0],
    [0, 0, 1],
], dtype=torch.float32).unsqueeze(0)


kornia_aff_1F = kornia.geometry.warp_affine(src_th, M1[:, :2], size, align_corners=False)
kornia_aff_1T = kornia.geometry.warp_affine(src_th, M2[:, :2], size, align_corners=True)
kornia_aff_2F = kornia.geometry.warp_affine(src_th, M2[:, :2], size*S, align_corners=False)
kornia_aff_2T = kornia.geometry.warp_affine(src_th, M2[:, :2], size*S, align_corners=True)
kornia_per_1F = kornia.geometry.warp_perspective(src_th, M1, size, align_corners=False)
kornia_per_1T = kornia.geometry.warp_perspective(src_th, M1, size, align_corners=True)
kornia_per_2F = kornia.geometry.warp_perspective(src_th, M2, size*S, align_corners=False)
kornia_per_2T = kornia.geometry.warp_perspective(src_th, M2, size*S, align_corners=True)

th_ups_1F = torch.nn.Upsample([*size], mode='bilinear', align_corners=False)(src_th)
th_ups_1T = torch.nn.Upsample([*size], mode='bilinear', align_corners=True)(src_th)
th_ups_2F = torch.nn.Upsample([*(size*S)], mode='bilinear', align_corners=False)(src_th)
th_ups_2T = torch.nn.Upsample([*(size*S)], mode='bilinear', align_corners=True)(src_th)

cv2_aff_1 = cv2.warpAffine(src_cv2, M1[0, :2].numpy(), size, flags=cv2.INTER_LINEAR)
cv2_aff_2 = cv2.warpAffine(src_cv2, M2[0, :2].numpy(), size*S, flags=cv2.INTER_LINEAR)
cv2_res_1 = cv2.resize(src_cv2, size)
cv2_res_2 = cv2.resize(src_cv2, size*S)
cv2_per_1 = cv2.warpPerspective(src_cv2, M1[0].numpy(), size, flags=cv2.INTER_LINEAR)
cv2_per_2 = cv2.warpPerspective(src_cv2, M2[0].numpy(), size*S, flags=cv2.INTER_LINEAR)

print()
print('=== Test 1: Identity ===')
print('Source tensor:')
print(src)
print()
print('  = Resize =')
print('   - OpenCV')
print(cv2_res_1)
print()
print('  = Upsample =')
print('   - Torch  (align_corners=False):')
print(th_ups_1F)
print()
print('   - Torch  (align_corners=True):')
print(th_ups_1T)
print()
print('  = Warp Affine =')
print('   - Kornia (align_corners=False):')
print(kornia_aff_1F)
print()
print('   - Kornia (align_corners=True):')
print(kornia_aff_1T)
print()
print('   - OpenCV:')
print(cv2_aff_1)
print()
print('  = Warp Perspective =')
print('   - Kornia (align_corners=False):')
print(kornia_per_1F)
print()
print('   - Kornia (align_corners=True):')
print(kornia_per_1T)
print()
print('   - OpenCV:')
print(cv2_per_1)

print()
print()

print()
print(f'=== Test 2: Scale {S:d}x ===')
print('Source tensor:')
print(src)
print()
print('  = Resize =')
print('   - OpenCV')
print(cv2_res_2)
print()
print('  = Upsample =')
print('   - Torch  (align_corners=False):')
print(th_ups_2F)
print()
print('   - Torch  (align_corners=True):')
print(th_ups_2T)
print()
print('  = Warp Affine =')
print('   - Kornia (align_corners=False):')
print(kornia_aff_2F)
print()
print('   - Kornia (align_corners=True):')
print(kornia_aff_2T)
print()
print('   - OpenCV:')
print(cv2_aff_2)
print()
print('  = Warp Perspective =')
print('   - Kornia (align_corners=False):')
print(kornia_per_2F)
print()
print('   - Kornia (align_corners=True):')
print(kornia_per_2T)
print()
print('   - OpenCV:')
print(cv2_per_2)

And here's the output (click to reveal)

Kornia version 0.5.11

=== Test 1: Identity ===
Source tensor:
tensor([[0., 2., 4.],
        [2., 4., 6.],
        [4., 6., 8.]])

  = Resize =
   - OpenCV
[[0. 2. 4.]
 [2. 4. 6.]
 [4. 6. 8.]]

  = Upsample =
   - Torch  (align_corners=False):
tensor([[[[0., 2., 4.],
          [2., 4., 6.],
          [4., 6., 8.]]]])

   - Torch  (align_corners=True):
tensor([[[[0., 2., 4.],
          [2., 4., 6.],
          [4., 6., 8.]]]])

  = Warp Affine =
   - Kornia (align_corners=False):
tensor([[[[0., 2., 4.],
          [2., 4., 6.],
          [4., 6., 8.]]]])

   - Kornia (align_corners=True):
tensor([[[[0., 1., 2.],
          [1., 2., 3.],
          [2., 3., 4.]]]])

   - OpenCV:
[[0. 2. 4.]
 [2. 4. 6.]
 [4. 6. 8.]]

  = Warp Perspective =
   - Kornia (align_corners=False):
tensor([[[[0., 1., 1.],
          [1., 4., 3.],
          [1., 3., 2.]]]])

   - Kornia (align_corners=True):
tensor([[[[0., 2., 4.],
          [2., 4., 6.],
          [4., 6., 8.]]]])

   - OpenCV:
[[0. 2. 4.]
 [2. 4. 6.]
 [4. 6. 8.]]



=== Test 2: Scale 2x ===
Source tensor:
tensor([[0., 2., 4.],
        [2., 4., 6.],
        [4., 6., 8.]])

  = Resize =
   - OpenCV
[[0.  0.5 1.5 2.5 3.5 4. ]
 [0.5 1.  2.  3.  4.  4.5]
 [1.5 2.  3.  4.  5.  5.5]
 [2.5 3.  4.  5.  6.  6.5]
 [3.5 4.  5.  6.  7.  7.5]
 [4.  4.5 5.5 6.5 7.5 8. ]]

  = Upsample =
   - Torch  (align_corners=False):
tensor([[[[0.0000, 0.5000, 1.5000, 2.5000, 3.5000, 4.0000],
          [0.5000, 1.0000, 2.0000, 3.0000, 4.0000, 4.5000],
          [1.5000, 2.0000, 3.0000, 4.0000, 5.0000, 5.5000],
          [2.5000, 3.0000, 4.0000, 5.0000, 6.0000, 6.5000],
          [3.5000, 4.0000, 5.0000, 6.0000, 7.0000, 7.5000],
          [4.0000, 4.5000, 5.5000, 6.5000, 7.5000, 8.0000]]]])

   - Torch  (align_corners=True):
tensor([[[[0.0000, 0.8000, 1.6000, 2.4000, 3.2000, 4.0000],
          [0.8000, 1.6000, 2.4000, 3.2000, 4.0000, 4.8000],
          [1.6000, 2.4000, 3.2000, 4.0000, 4.8000, 5.6000],
          [2.4000, 3.2000, 4.0000, 4.8000, 5.6000, 6.4000],
          [3.2000, 4.0000, 4.8000, 5.6000, 6.4000, 7.2000],
          [4.0000, 4.8000, 5.6000, 6.4000, 7.2000, 8.0000]]]])

  = Warp Affine =
   - Kornia (align_corners=False):
tensor([[[[0.0000, 0.7109, 1.7266, 2.7422, 2.2344, 0.2031],
          [0.7109, 1.7500, 3.0000, 4.2500, 3.3516, 0.3047],
          [1.7266, 3.0000, 4.2500, 5.5000, 4.2109, 0.3828],
          [2.7422, 4.2500, 5.5000, 6.7500, 5.0703, 0.4609],
          [2.2344, 3.3516, 4.2109, 5.0703, 3.7812, 0.3438],
          [0.2031, 0.3047, 0.3828, 0.4609, 0.3438, 0.0313]]]])

   - Kornia (align_corners=True):
tensor([[[[0.0000, 1.0000, 2.0000, 3.0000, 4.0000, 2.0000],
          [1.0000, 2.0000, 3.0000, 4.0000, 5.0000, 2.5000],
          [2.0000, 3.0000, 4.0000, 5.0000, 6.0000, 3.0000],
          [3.0000, 4.0000, 5.0000, 6.0000, 7.0000, 3.5000],
          [4.0000, 5.0000, 6.0000, 7.0000, 8.0000, 4.0000],
          [2.0000, 2.5000, 3.0000, 3.5000, 4.0000, 2.0000]]]])

   - OpenCV:
[[0.  1.  2.  3.  4.  2. ]
 [1.  2.  3.  4.  5.  2.5]
 [2.  3.  4.  5.  6.  3. ]
 [3.  4.  5.  6.  7.  3.5]
 [4.  5.  6.  7.  8.  4. ]
 [2.  2.5 3.  3.5 4.  2. ]]

  = Warp Perspective =
   - Kornia (align_corners=False):
tensor([[[[0.0000, 0.2500, 1.0000, 1.7500, 1.0000, 0.0000],
          [0.2500, 1.0000, 2.5000, 4.0000, 2.2500, 0.0000],
          [1.0000, 2.5000, 4.0000, 5.5000, 3.0000, 0.0000],
          [1.7500, 4.0000, 5.5000, 7.0000, 3.7500, 0.0000],
          [1.0000, 2.2500, 3.0000, 3.7500, 2.0000, 0.0000],
          [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]]]])

   - Kornia (align_corners=True):
tensor([[[[0.0000, 1.0000, 2.0000, 3.0000, 4.0000, 2.0000],
          [1.0000, 2.0000, 3.0000, 4.0000, 5.0000, 2.5000],
          [2.0000, 3.0000, 4.0000, 5.0000, 6.0000, 3.0000],
          [3.0000, 4.0000, 5.0000, 6.0000, 7.0000, 3.5000],
          [4.0000, 5.0000, 6.0000, 7.0000, 8.0000, 4.0000],
          [2.0000, 2.5000, 3.0000, 3.5000, 4.0000, 2.0000]]]])

   - OpenCV:
[[0.  1.  2.  3.  4.  2. ]
 [1.  2.  3.  4.  5.  2.5]
 [2.  3.  4.  5.  6.  3. ]
 [3.  4.  5.  6.  7.  3.5]
 [4.  5.  6.  7.  8.  4. ]
 [2.  2.5 3.  3.5 4.  2. ]]

On the blue dots diagram (click to reveal)

Specifically when resizing, OpenCV and PyTorch don’t extrapolate beyond the source image boundaries (in contrast to warping, where OpenCV does extrapolate). Instead the interpolation indices get clipped to the image boundary, so when resizing in PyTorch and OpenCV we actually end up with data like this: better-blue This is to explain my earlier comment that it didn’t match the blue dots diagram reference earlier in this issue. The original blue dots diagram is more appropriate for discussions on warping, since you choose how to pad the original image for warping, and thus can extrapolate beyond the original image boundaries.

Multihuntr on Oct 4, 2021

@edgarriba This should not be closed by the bot, since the bug won’t go away if no one has posted anything new.

pmeier on Dec 26, 2020