zarr-python: Passing dask array to zarr leads to a TypeError
Minimal, reproducible code sample, a copy-pastable example if possible
import numpy as np
import zarr
import os
import dask.array as da
from ome_zarr.io import parse_url
from ome_zarr.writer import write_image
path = "test_dask_image.zarr"
os.mkdir(path)
mean_val = 10
rng = np.random.default_rng(0)
data = rng.poisson(mean_val, size=(50, 128, 128)).astype(np.uint8)
data = da.from_array(data) # <--- !!!THIS IS WHAT BREAKS THE EXAMPLE!!!
# write the image data
store = parse_url(path, mode="w").store
root = zarr.group(store=store)
write_image(image=data, group=root, chunks=(1, 128, 128), axes="zyx")
root.attrs["omero"] = {
"channels": [{
"color": "00FFFF",
"window": {"start": 0, "end": 20},
"label": "random",
"active": True,
}]
}
Problem description
Get an error when trying to write a dask array to a zarr file. I took the proposed example from https://github.com/ome/ome-zarr-py/pull/121 and wrapped the numpy array in a dask array. It appears that the dask array function astype doesn’t support the parameter “order”. The example code works when using a numpy array for me. Here’s the resulting traceback:
Traceback (most recent call last):
File "/home/sutherland/git/xray_napari/write_zarr_with_dask_failure.py", line 20, in <module>
write_image(image=data, group=root, chunks=(1, 128, 128), axes="zyx")
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/ome_zarr/writer.py", line 178, in write_image
write_multiscale(image, group, chunks=chunks, fmt=fmt, axes=axes)
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/ome_zarr/writer.py", line 105, in write_multiscale
group.create_dataset(str(path), data=dataset, chunks=chunks)
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/zarr/hierarchy.py", line 808, in create_dataset
return self._write_op(self._create_dataset_nosync, name, **kwargs)
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/zarr/hierarchy.py", line 661, in _write_op
return f(*args, **kwargs)
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/zarr/hierarchy.py", line 824, in _create_dataset_nosync
a = array(data, store=self._store, path=path, chunk_store=self._chunk_store,
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/zarr/creation.py", line 377, in array
z[...] = data
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/zarr/core.py", line 1224, in __setitem__
self.set_basic_selection(selection, value, fields=fields)
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/zarr/core.py", line 1319, in set_basic_selection
return self._set_basic_selection_nd(selection, value, fields=fields)
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/zarr/core.py", line 1610, in _set_basic_selection_nd
self._set_selection(indexer, value, fields=fields)
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/zarr/core.py", line 1682, in _set_selection
self._chunk_setitems(lchunk_coords, lchunk_selection, chunk_values,
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/zarr/core.py", line 1871, in _chunk_setitems
cdatas = [self._process_for_setitem(key, sel, val, fields=fields)
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/zarr/core.py", line 1871, in <listcomp>
cdatas = [self._process_for_setitem(key, sel, val, fields=fields)
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/zarr/core.py", line 1924, in _process_for_setitem
chunk = value.astype(self._dtype, order=self._order, copy=False)
File "/home/sutherland/anaconda3/envs/napari2/lib/python3.9/site-packages/dask/array/core.py", line 2086, in astype
raise TypeError(
TypeError: astype does not take the following keyword arguments: ['order']
Version and installation information
Please provide the following:
- Value of
zarr.__version__
: 2.10.3 - Value of
numcodecs.__version__
: 0.9.1 - Version of Python interpreter: 3.9.7
- Operating system (Linux/Windows/Mac): RHEL 8
- How Zarr was installed (e.g., “using pip into virtual environment”, or “using conda”): using pip into conda env
Also, if you think it might be relevant, please provide the output from pip freeze
or
conda env export
depending on which was used to install Zarr.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 24 (12 by maintainers)
I’m +1 on redirecting everyone back to use the
dask.array.to_zarr
method instead of making additional changes here.Thanks for the info, @michael-sutherland. Looking at the stacktrace again, any objections to migrating this to github.com/ome/ome-zarr-py ?
cc: @sbesson @will-moore