VirtualiZarr: Tiff test error

/home/ubuntu/anaconda3/envs/pangeo/lib/python3.11/site-packages/virtualizarr/readers/tiff.py:47: UserWarning: storage_options have been dropped from reader_options as they are not supported by kerchunk.tiff.tiff_to_zarr
  warnings.warn(
Traceback (most recent call last):
  File "/home/ubuntu/data/model-framework/test_vz.py", line 20, in <module>
    vds = open_virtual_dataset("s3://bananasplits/rasters/fleagle.tif", reader_options={'storage_options': aws_credentials})
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/pangeo/lib/python3.11/site-packages/virtualizarr/backend.py", line 200, in open_virtual_dataset
    vds = backend_cls.open_virtual_dataset(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/pangeo/lib/python3.11/site-packages/virtualizarr/readers/tiff.py", line 55, in open_virtual_dataset
    refs = extract_group(refs, group)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/anaconda3/envs/pangeo/lib/python3.11/site-packages/virtualizarr/translators/kerchunk.py", line 55, in extract_group
    raise ValueError(
ValueError: Multiple HDF Groups found. Must specify group= keyword to select one of []

Installed vz half an hour ago, python 3.11.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 25 (14 by maintainers)

Most upvoted comments

To jump in on this - here’s a set of GeoTIFFs I’m working on virtualizing: SENTINEL1 Sigma Nought (SIG0) Backscatter in 20 meter resolution.

I’ve been following a manual approach with tifffile following this tutorial and the lessons from this issue https://github.com/fsspec/kerchunk/issues/78, but the lack of access to intra-file chunks makes it a nonstarter for our usecase unfortunately. We’re fitting models to multi-year timeseries stacks per pixel, which requires loading the entire stack of images per 15000x15000 tile into memory when you can’t access sub-file chunks. Too much for our beefy machines. I can build the virtual zarr just fine and use it to load data without a problem, but this specific processing case is important and just doesn’t work well right now.

Note the nonstandard blocking of 5x15000. I’ve heard tell from coworkers that there may be some images in there with different blocking but I haven’t run any checks to confirm that yet. I do wonder if this would cause some issues when using the proposed TIF reader if true.

I admire all the hard work being done here to enable big geodata workflows, huge thanks to everyone involved 😃

thank you @RichardScottOZ! what timezone are you in? I know there’s a lot of shared interest right now on Virtual approaches with TIFFs including you, me (e.g., https://github.com/maxrjones/why-virtualize-geotiff), @mdsumner, and @norlandrhagen. It could be fun to have a bug smashing session on the TIFF reader next week if you’re interested

FYI we can (and it sounds like we should) have multiple TIFF readers for virtualizarr. After #498 implementing one of these readers will be as simple as writing one function like reader_func(path: str) -> ManifestStore:, so we can and should have multiple to serve all possible requirements.

We’re fitting models to multi-year timeseries stacks per pixel, which requires loading the entire stack of images per 15000x15000 tile into memory when you can’t access sub-file chunks. Too much for our beefy machines. I can build the virtual zarr just fine and use it to load data without a problem, but this specific processing case is important and just doesn’t work well right now.

This sounds like a scenario where you’re better off rechunking your data to align more with your access pattern (i.e. contiguous in time; chunked in space) and then storing it as native (not virtual Zarr).

I’m currently debating whether it would even be better to develop the async_tiff reader separately from the VirtualiZarr repo

I would be fine with that. But we should have some default TIFF reader somewhere because it’s such a widely-used format.

Some features found in TIFF that I have been struggling to represent in virtual references are bitorder reversal, packed integers, float24 and complex integer data types, color profiles, variable JPEG tables across TIFF pages, and incomplete chunks (cropped tiles and last strips).

I think this question is less about parsing TIFF and more about how to represent the parsed metadata in virtual references?

Yes, but mostly it’s actually a zarr issue rather than a virtualizarr issue (requiring new codecs or data types for example). It’s easy to represent any codec or data type in virtualizarr, iff it is actually supported by zarr.

I agree that there’s scope for multiple TIFF readers. tifffile has been around longer, is in much wider use, and seems to be very stable, so it’s probably the better default to use. But judging from other obstore performance improvements (https://github.com/zarr-developers/zarr-python/pull/1661#issuecomment-2780104437) I think there’s still potential in an opt-in async-tiff backend (they each use the same Rust IO code under the hood).

Does the async_tiff approach handle TIFF-like formats (ImageJ, LSM, NDPI, etc), volumetric tiles, sparse segments, higher dimensional datasets, JPEG tables?

The current API of async-tiff is quite minimal and mostly consists of parsing and then exposing raw IFD metadata to the user.

I would say that TIFF-like formats that are not actually TIFF are not in scope. It is able to parse the JPEG tables from the IFD metadata though.

Some features found in TIFF that I have been struggling to represent in virtual references are bitorder reversal, packed integers, float24 and complex integer data types, color profiles, variable JPEG tables across TIFF pages, and incomplete chunks (cropped tiles and last strips).

I think this question is less about parsing TIFF and more about how to represent the parsed metadata in virtual references? In that case those questions go back to @maxrjones and @TomNicholas, as I haven’t been following that side of the work stream. I’ve only been working on the low-level format parsing.

FYI we can (and it sounds like we should) have multiple TIFF readers for virtualizarr. After https://github.com/zarr-developers/VirtualiZarr/issues/498 implementing one of these readers will be as simple as writing one function like reader_func(path: str) -> ManifestStore:, so we can and should have multiple to serve all possible requirements.

+1. I’m currently debating whether it would even be better to develop the async_tiff reader separately from the VirtualiZarr repo rather than continuing to build it as a refactor of TIFFVirtualBackend as in https://github.com/zarr-developers/VirtualiZarr/pull/524. Here are my reasons for considering building it out separately (currently experimenting in https://github.com/maxrjones/virtual-tiff):

  • If there is a single canonical TIFF reader, it might make more sense to base it off TIFFFile as a more established library than async_tiff
  • I’d like to be really comprehensive in testing, which might be discouraged considering the VirtualiZarr’s primary objective is not readers. As one example, I’m currently downloading and using all of GDAL’s test TIFF files (thanks to Michael for the pointer to these)
  • Developing the async_tiff reader separately would speed up tests all around
  • VirtualiZarr is approaching a stable API while also getting a lot more engagement, so it might be time in general to consider splitting off readers to achieve a more sustainable maintenance model

This sounds like a scenario where you’re better off rechunking your data to align more with your access pattern (i.e. contiguous in time; chunked in space) and then storing it as native (not virtual Zarr).

I absolutely agree, and this is where we’re slowly headed, but this is a PB-scale dataset that plenty of operational workflows depend on, so it’s going to take a while before that’s done unfortunately.

Virtualizing it would be a really nice in-between step, and it feels so close if only we could represent those tiff chunks.

I’ll dig up other large files that aren’t COGs but famous ones are GEBCO and IBCSO:

https://www.gebco.net/data_and_products/gridded_bathymetry_data/ (the tiff link is a directory that is the file in a zip, so it’s weird to have that as a canon url and while you can stream it it doesn’t work in some places)

https://doi.pangaea.de/10.1594/PANGAEA.937574?format=html#download (https://download.pangaea.de/dataset/937574/files/IBCSO_v2_ice-surface_WGS84.tif)