openseadragon: Tile alignment bug (possible rounding error?)

As described in a comment on another PR, I have noticed a problem that causes tiles to get out of alignment on certain images at certain zoom images. The linked comment contains screen shots and details on how to reproduce.

I have done a git bisect and have found that the problem seems to have been introduced by this commit, which was first introduced in v2.4.0: https://github.com/openseadragon/openseadragon/commit/075a37ce8c7f82651f823d7f0c0bebce4214f058

I can see that tinkering with the numbers in getTileBounds in tilesource.js has some impact on the behavior – I suspect there’s some kind of rounding error putting tiles out of alignment in certain situations… but it’s a bit over my head to figure out what the solution might be!

Any ideas, @iangilman?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 38 (20 by maintainers)

Commits related to this issue

Most upvoted comments

My pleasure – I appreciate the great community effort!

@demiankatz Great! Thank you for confirming, and for working through this issue 😊

@ruven @iangilman I’ve tested the new code (https://github.com/openseadragon/openseadragon/commit/ae500afbcb143a8af49f0f8bf7c61d8f3e1ae56f) and it seems to work for us. Our IIIF server always generates “scaleFactors: [1, 2, 4, 8, 16]”, so “this.levelSizes” will be set only when the sizes array has 5 elements (which depends on the original resolution). Even in that case, I’ve seen no problems: no misalignment, no negative values in the request URL. Can be tested with this: info.json I hope this will fix it for everyone. Thank you, ruven.

On the question of the use of the sizes array:

The use of the sizes array was to fix problems related to differences in rounding between OSD and different image codecs / image servers when requesting the sub-resolutions of an image (original discussion: https://github.com/openseadragon/openseadragon/issues/2321). In addition, for image formats such as multi-resolution pyramid TIFF or JPEG2000, these sub-resolutions exist “physically” in the image format, so aligning to these exact dimensions makes processing faster and more efficient. So, it’s definitely worth using an exact list of resolution sizes if this is available.

In many cases, for example when using IIIF servers such as IIPImage or Canteloupe, this list will indeed consist of the sizes of the various resolution levels. However, as we can see in some of the problem examples, the sizes array does not necessarily contain a list of sub-resolutions and the API spec doesn’t, in any case, explicitly state that this array should be used in this way. The API spec only states that the sizes array should contain the “preferred height and width combinations for representations of the full image”. The use of sizes is, therefore, in reality a bit of hack to fix an ambiguity in the API itself (https://github.com/IIIF/api/issues/2232).

So, the solution here is not to discard sizes, but to make the checks much stricter. My original pull request https://github.com/openseadragon/openseadragon/pull/2337 assumes that the sizes list is well-behaved, but clearly this is not a valid assumption. So I’ve re-factored the code (https://github.com/openseadragon/openseadragon/pull/2710/commits/ae500afbcb143a8af49f0f8bf7c61d8f3e1ae56f) to make it much stricter and to make it check each size against the given scalingFactors. With this patch, the use of sizes is skipped in all the cited problem examples. This should resolve issue https://github.com/openseadragon/openseadragon/issues/2618 and fix the problem with the 2 images referred to in https://github.com/openseadragon/openseadragon/issues/2701#issuecomment-2792268149. Please take a look and let me know what you all think. This change may possibly also fix the issues in https://github.com/openseadragon/openseadragon/issues/2691 and https://github.com/openseadragon/openseadragon/issues/2452.

@iangilman @demiankatz Problem started here: https://github.com/openseadragon/openseadragon/commit/0c358c140dae8bcd5cbc568ef6236a840af84efe and was published with v4.1.0. The comment of the commit is “Use resolution level dimensions provided in the info.json “sizes” field to determine tile sizes as well as the number of tiles that exist at a particular resolution.” I don’t think you can use the sizes array to calculate tile sizes since it is only for scaled full(!) images.

https://iiif.io/api/image/3.0/compliance/#5-level-0-compliance

A level 0 compliant image server may specify scaleFactors and/or width and height values for tiles in the Image Information response. At level 0 compliance, a service is only required to deliver images of sizes computed using the scaling factors declared in the Image Information response. If specified they should be interpreted with the following special meanings: scaleFactors - only the specified scaling factors are supported width, height within tiles - clients should request only regions that correspond to output tiles of the specified dimensions

A level 0 compliant image server may specify sizes to indicate specific width and height values for sizes available for the full region. If a client requests a combination of size and region outside these parameters then the image server may reject the request with an error.

I think the two has been combined by mistake and it is kind of random whether it works or not: depending on the number of scaleFactors and elements in the sizes array for example.

EDIT: I think it all started here: https://github.com/openseadragon/openseadragon/issues/2321#issuecomment-1507624044 Maybe this is why the sizes array was used… But it should have been much easier… in the tiles array, when the height is missing, it simply means to use the same as the width, so it’s a square: https://iiif.io/api/image/3.0/#54-tiles

The height in pixels of the predefined tiles to be requested, given as an integer. If it is not specified in the JSON, then it defaults to the same as width, resulting in square tiles.

Level 1+ services should support w,h requests if they’re compliant:

https://iiif.io/api/image/2.0/compliance/#size

Even if Cantaloupe fix their sizes bug, this particular request will never work correctly as there is no way to get a height of 1024 if you only specify the width and if it is set to an integer value. If you set the width to 181 you get height = 1021 and if you set width to 182 you get height = 1027. Their use of an extra-large tile size of 1024 means that every rounding error gets exaggerated.

Or change the OSD code to include height as well? Perhaps something is in the info.json that would indicate a level 0 with width only, so that case would be handled separately?

You could do something like

            } else if (isLevel0) {
                iiifSize = iiifSizeW + ",";
            } else {
                iiifSize = iiifSizeW + "," + iiifSizeH;
            }

But this risks to break a number of existing v2 installations which don’t expect width,height tile requests.

The response headers say:

x-powered-by: Cantaloupe/5.0.5

So perhaps it’s a bug there?

I did not realize that Wellcome was using Cantaloupe, but all signs do now point to this being some kind of Cantaloupe bug. I used the Wellcome example in my ticket because it’s from a manifest that’s commonly used as a Universal Viewer example, but I first noticed the problem when working with Villanova manifests. Villanova currently uses Cantaloupe 5.0.6, so the issue is definitely still present there. I notice that Cantaloupe 5.0.7 was released very recently, and it includes a number of bug fixes. Next week, I will see if I can replicate the problem in my test environment, then upgrade to the latest Cantaloupe to see if that makes a difference.

I do think there may also be value in investigating ways to make the OSD requests more specific when feasible, to reduce the chances of ambiguity and inconsistent behavior – but I understand that the “when feasible” part is important, and perhaps there isn’t a reliable way to make things better for this case without potentially creating new problems.

I’ll report back when I have more information – thanks for the discussion and investigation, everyone!

So, the only way to fix this is to upgrade the server to IIIF v3

Or change the OSD code to include height as well? Perhaps something is in the info.json that would indicate a level 0 with width only, so that case would be handled separately?

Canteloupe supports both 2 and 3 iirc, but some sites are probably stuck on 2 (for instance, not having the bandwidth to upgrade their manifests to v3).

Just to note the reason for only specifying the width for v2 is due to Canonical URI Syntax from: https://iiif.io/api/image/2.1/#canonical-uri-syntax. This is particularly important for a level 0 IIIF image which would only thave the width only image.

So, the only way to fix this is to upgrade the server to IIIF v3 because it’s impossible to get a tile height of exactly 1024 for this particular image when specifying only the width.

The response headers say:

x-powered-by: Cantaloupe/5.0.5

That’s strange - Canteloupe usually correctly populates the sizes field in the info.json.

The response headers say:

x-powered-by: Cantaloupe/5.0.5

So perhaps it’s a bug there?

Just to note the reason for only specifying the width for v2 is due to Canonical URI Syntax from: https://iiif.io/api/image/2.1/#canonical-uri-syntax. This is particularly important for a level 0 IIIF image which would only thave the width only image.

If we just consider the image request in isolation, the server is in fact responding the way it should.

The image has a native size of 2411x3372 according to the info.json file. The request asks for the region 2048,0,363,2048 from this full resolution image, which means a region from x=2048 of size 363 in width (so up to the right-hand edge of the image) and 2048 in height starting from the top (y=0). So, an image of size 363x2048.

The IIIF request then says it wants this 363x2048 image to be scaled to 182 pixels horizontally. The height calculation is (182/363) * 2048 = 1026.82 which rounds to 1027 and this is what is correctly returned by the server. By the way, if you ask for a width of 181 instead, the height = 1021.18 (so still not 1024) and the server does indeed correctly return a height of 1021:

https://iiif.wellcomecollection.org/image/b18035723_0003.JP2/2048,0,363,2048/181,/0/default.jpg

So, is the problem on the OSD-side? Not really. The problem does indeed come from a rounding issue, but the problem is server-side and related to the info.json file (I don’t know what IIIF server they are using - it’s not IIPImage in any case). If you take a look at this info.json, you’ll notice it doesn’t provide sufficient information on the available sizes of each resolution. In the Tiles section, it says there are 6 resolution levels available with scaleFactors from 1 to 32. However, the sizes field only contains 4 sets of dimensions and these don’t correspond to the 6 resolution levels. OSD, therefore, cannot use these values and, therefore, has to calculate the sizes of these levels itself. If these resolution sizes were in the info.json, OSD would have used these to determine the exact tile sizes. The final rounding error is then compounded by the fact they are using a very large tile size of 1024x1024.

it’s definitely possible to provide both dimensions; see the w,h option in the size section of the specification (link is to the image 3.0 spec, but the option is included in 1.0 and 2.0 as well)

In fact the OSD function getTileURL() for IIIF includes both the width and height if it’s version 3, but not if it’s 1 or 2 (the server is only v2):

            } else if (this.version === 3) {
                iiifSize = iiifSizeW + "," + iiifSizeH;
            } else {
                iiifSize = iiifSizeW + ",";
            }

If I remember correctly, this is coded this way because the use of a single dimension for tile requests is stipulated in the spec somewhere.

So, the best solution is to get the Welcome Collection to upgrade to IIIF v3 and/or fix the info.json files. Alternatively, an OSD solution when using IIIF version 2 or lower, would be to switch between specifying only width or height depending on which is larger.

For example, if you request a specific height rather than width for this same image you get the correct size of 182x1024.: https://iiif.wellcomecollection.org/image/b18035723_0003.JP2/2048,0,363,2048/,1024/0/default.jpg