VirtualiZarr: Tiff test error
/home/ubuntu/anaconda3/envs/pangeo/lib/python3.11/site-packages/virtualizarr/readers/tiff.py:47: UserWarning: storage_options have been dropped from reader_options as they are not supported by kerchunk.tiff.tiff_to_zarr
warnings.warn(
Traceback (most recent call last):
File "/home/ubuntu/data/model-framework/test_vz.py", line 20, in <module>
vds = open_virtual_dataset("s3://bananasplits/rasters/fleagle.tif", reader_options={'storage_options': aws_credentials})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/pangeo/lib/python3.11/site-packages/virtualizarr/backend.py", line 200, in open_virtual_dataset
vds = backend_cls.open_virtual_dataset(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/pangeo/lib/python3.11/site-packages/virtualizarr/readers/tiff.py", line 55, in open_virtual_dataset
refs = extract_group(refs, group)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/pangeo/lib/python3.11/site-packages/virtualizarr/translators/kerchunk.py", line 55, in extract_group
raise ValueError(
ValueError: Multiple HDF Groups found. Must specify group= keyword to select one of []
Installed vz half an hour ago, python 3.11.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 25 (14 by maintainers)
To jump in on this - here’s a set of GeoTIFFs I’m working on virtualizing: SENTINEL1 Sigma Nought (SIG0) Backscatter in 20 meter resolution.
I’ve been following a manual approach with
tifffilefollowing this tutorial and the lessons from this issue https://github.com/fsspec/kerchunk/issues/78, but the lack of access to intra-file chunks makes it a nonstarter for our usecase unfortunately. We’re fitting models to multi-year timeseries stacks per pixel, which requires loading the entire stack of images per 15000x15000 tile into memory when you can’t access sub-file chunks. Too much for our beefy machines. I can build the virtual zarr just fine and use it to load data without a problem, but this specific processing case is important and just doesn’t work well right now.Note the nonstandard blocking of 5x15000. I’ve heard tell from coworkers that there may be some images in there with different blocking but I haven’t run any checks to confirm that yet. I do wonder if this would cause some issues when using the proposed TIF reader if true.
I admire all the hard work being done here to enable big geodata workflows, huge thanks to everyone involved 😃
thank you @RichardScottOZ! what timezone are you in? I know there’s a lot of shared interest right now on Virtual approaches with TIFFs including you, me (e.g., https://github.com/maxrjones/why-virtualize-geotiff), @mdsumner, and @norlandrhagen. It could be fun to have a bug smashing session on the TIFF reader next week if you’re interested
FYI we can (and it sounds like we should) have multiple TIFF readers for virtualizarr. After #498 implementing one of these readers will be as simple as writing one function like
reader_func(path: str) -> ManifestStore:, so we can and should have multiple to serve all possible requirements.This sounds like a scenario where you’re better off rechunking your data to align more with your access pattern (i.e. contiguous in time; chunked in space) and then storing it as native (not virtual Zarr).
here’s a few COGs that I would expect to work, I thought I was just doing something wrong (but also didn’t pursue very deeply)
https://projects.pawsey.org.au/idea-gebco-tif/GEBCO_2024.tif (4Gb)
https://github.com/mdsumner/rema-ovr/raw/refs/heads/main/rema_mosaic_1km_v2.0_filled_cop30_dem.tif (60Mb)
https://e84-earth-search-sentinel-data.s3.us-west-2.amazonaws.com/sentinel-2-c1-l2a/55/G/EN/2025/3/S2C_T55GEN_20250324T000834_L2A/TCI.tif (250Mb)
https://data.source.coop/ausantarctic/ghrsst-mur-v2/2025/03/31/20250331090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1_analysed_sst.tif (300Mb)
https://github.com/mdsumner/ibcso-cog/raw/main/IBCSO_v2_ice-surface_cog.tif (220Mb)
I would also add the autotest suite from GDAL to really get a comprehensive try at file vagaries
I’ve listed all current “\.tif$” files here (prepend ‘https://raw.githubusercontent.com/OSGeo/gdal/refs/heads/master/autotest’ to get full path)
https://gist.githubusercontent.com/mdsumner/29b22ece80c829ae4aefbecbf4eef531/raw/a8ea6f13f8c96b8434b7fc3ae0629814914135c0/autotest_tif.txt
I would be fine with that. But we should have some default TIFF reader somewhere because it’s such a widely-used format.
Yes, but mostly it’s actually a zarr issue rather than a virtualizarr issue (requiring new codecs or data types for example). It’s easy to represent any codec or data type in virtualizarr, iff it is actually supported by zarr.
I agree that there’s scope for multiple TIFF readers. tifffile has been around longer, is in much wider use, and seems to be very stable, so it’s probably the better default to use. But judging from other obstore performance improvements (https://github.com/zarr-developers/zarr-python/pull/1661#issuecomment-2780104437) I think there’s still potential in an opt-in async-tiff backend (they each use the same Rust IO code under the hood).
The current API of async-tiff is quite minimal and mostly consists of parsing and then exposing raw IFD metadata to the user.
I would say that TIFF-like formats that are not actually TIFF are not in scope. It is able to parse the JPEG tables from the IFD metadata though.
I think this question is less about parsing TIFF and more about how to represent the parsed metadata in virtual references? In that case those questions go back to @maxrjones and @TomNicholas, as I haven’t been following that side of the work stream. I’ve only been working on the low-level format parsing.
+1. I’m currently debating whether it would even be better to develop the
async_tiffreader separately from the VirtualiZarr repo rather than continuing to build it as a refactor of TIFFVirtualBackend as in https://github.com/zarr-developers/VirtualiZarr/pull/524. Here are my reasons for considering building it out separately (currently experimenting in https://github.com/maxrjones/virtual-tiff):TIFFFileas a more established library thanasync_tiffasync_tiffreader separately would speed up tests all aroundI absolutely agree, and this is where we’re slowly headed, but this is a PB-scale dataset that plenty of operational workflows depend on, so it’s going to take a while before that’s done unfortunately.
Virtualizing it would be a really nice in-between step, and it feels so close if only we could represent those tiff chunks.
I’ll dig up other large files that aren’t COGs but famous ones are GEBCO and IBCSO:
https://www.gebco.net/data_and_products/gridded_bathymetry_data/ (the tiff link is a directory that is the file in a zip, so it’s weird to have that as a canon url and while you can stream it it doesn’t work in some places)
https://doi.pangaea.de/10.1594/PANGAEA.937574?format=html#download (https://download.pangaea.de/dataset/937574/files/IBCSO_v2_ice-surface_WGS84.tif)