VirtualiZarr: open_virtual_dataset fails to open tiffs
open_virtual_dataset says it supports TIFF files however it currently fails and is lacking test coverage. The issue starts here where the kerchunk reference to a zarr array is treated as an HDF reference that has one or more groups.
Code to reproduce
from pathlib import Path
import numpy as np
from PIL import Image
from virtualizarr import open_virtual_dataset
from virtualizarr.backend import FileType
def write_random_tiff(file_path):
array = np.random.randint(0, 255, (128, 128), dtype=np.uint8)
img = Image.fromarray(array)
img.save(file_path)
file_path = Path.cwd() / "test.tif"
write_random_tiff(file_path)
open_virtual_dataset(str(file_path), filetype=FileType.tiff)
Throws:
File virtualizarr/translators/kerchunk.py:46, in extract_group(vds_refs, group)
ValueError: Multiple HDF Groups found. Must specify group= keyword to select one of []
virtualizarr = 1.1.0
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 3
- Comments: 21 (14 by maintainers)
Yes a dataarray is a better fit. I just need the ManifestArray object.
It does not at the moment.
That’s correct, and
write_fsspecis a method oftifffile.ZarrTiffStore. It’s designed to write to a file, butkerchunk.tiff.tiff_to_zarrwrites it to an in-memory text buffer and then converts it back to json.@mdsumner That file has two arrays with the same shape and dimensions.
tifffile.ZarrTiffStore.write_fsspecfor better or worse treats them as a single array with two chunks, one for each image, which fits better in axarray.DataArray. The example we need is one where a TIFF has multiple arrays of different shapes, dimensions, or coords (and probably keys for each array that hopefullytifffilecan parse), which would need to be wrapped in axarray.Datasetorxarray.DataTree. OR we need an example wheretifffile.ZarrTiffStore.write_fsspecunderstands a TIFF spec and places references to matching arrays (same shape and dimensions) under different zarr groups.Sounds good. Right now I can help most with developing tests but I’ll take a look at some of the other VirtualiZarr readers and try to get a grasp of what they ultimately produce.
Okay I tried an using an approach like
but ended up down a rabbit hole because a)
tifffiledoesn’t appear to support zarr-python v3 yet and b) determining the actual byte ranges is still tricky - inkerchunk.tiff.tiff_to_zarrI think it happens in thestore.write_fsspecstep.EDIT: I posted where I got to in #295 in case it’s ever of any use.
I think we should try to get the tiff reader working using an implementation that relies on the kerchunk reader first, then as a follow-up think about if we can eliminate kerchunk from that process.
Ah sorry about this @elyall !
We do have some rudimentary tests for tiff but as the tiff reader is just a thin wrapper around the kerchunk tiff reader I (naively) expected this to “just work”. I agree that we should remedy that with better test coverage.
Thanks for the reproducible example, I’ll have a look at what might be going on now.