VirtualiZarr: `to_icechunk` doesn't have a `group` option

This signature

https://github.com/zarr-developers/VirtualiZarr/blob/fcdd5e455a6054b5671659924b6b0e2cc93757ff/virtualizarr/writers/icechunk.py#L25-L27

should include a group option to allow you to put the dataset somewhere other than at the root of the store hierarchy.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 21 (10 by maintainers)

Commits related to this issue

Most upvoted comments

Or shall I make group kw-only as well?

I personally would also make group keyword-only (if we were to rewrite Dataset.to_zarr I’d also vote for making all the parameters other than store keyword-only):

dataset_to_icechunk(ds, store, group=x) is much harder to misunderstand compared to

dataset_to_icechunk(ds, store, x) (I’m intentionally choosing a bad variable name to make my point)

Thank you. I agree. I’ll take that route unless someone strongly advises otherwise.

@chuckwondo would you be interested in opening another PR that adds the group option to the to_icechunk method in the VirtualiZarrDatasetAccessor:

Sure!

Or shall I make group kw-only as well?

I personally would also make group keyword-only (if we were to rewrite Dataset.to_zarr I’d also vote for making all the parameters other than store keyword-only):

dataset_to_icechunk(ds, store, group=x)

is much harder to misunderstand than

dataset_to_icechunk(ds, store, x)

(I’m intentionally choosing a bad variable name to make my point)

I’d like to take this one, if nobody else has claimed it.

Thanks for the thoughtful Qs @chuckwondo. As this is orthogonal to the addition of the group kwarg I would prefer that we just plumb it through to the accessor for now (else right now the API is unclear + inconsistent), and I’ve raised #389 to discuss the general design choice of using accessors at all.

Excellent. Thanks for starting a separate discussion. I’ll add my 2 cents over there.

In the meantime, I’ve plumbed things through as requested, in PR #391

I think changing it to _dataset_to_icechunk would be worthwhile.

@chuckwondo would you be interested in opening another PR that adds the group option to the to_icechunk method in the VirtualiZarrDatasetAccessor: https://github.com/zarr-developers/VirtualiZarr/blob/e3787c55cd242d83257d2bc0ab819aab56dda3cb/virtualizarr/accessor.py#L42-L82

This is a convenience method that allows users to use code like:

vds.virtualize.to_icechunk(store, group=group))

rather than needing to call dataset_to_icehunk().

I think the best way to test this new code would be to modify the test you previously parameterized to use the to_icechunk() function on the accessor rather than calling dataset_to_icechunk because this will only add code coverage: https://github.com/zarr-developers/VirtualiZarr/blob/e3787c55cd242d83257d2bc0ab819aab56dda3cb/virtualizarr/tests/test_writers/test_icechunk.py#L42-L80

Offhand it seems like making append_dim, last_updated_at, and group (to be added) might be better as kw-only args, but I suspect that would represent a breaking change, so adding group as an optional parameter at the end seems like it would play better, but it feels like that’s starting to get messy because if a caller wants only to set group and not the other 2 optionals, they are forced to pass 2 Nones for those positional parameters

for reference, if group would also be a positional-or-keyword (i.e. not positional-only) parameter with a default – like append_dim / last_updated_at – it would be possible to simply use group as a keyword argument:

dataset_to_icechunk(ds, store, group=...)

(so you don’t have to make it a keyword-only parameter).

That said, keyword-only parameters allow making API use more uniform, which especially for optional parameters really improves the readability of any function calls (in my opinion, at least).

Ideally do whatever xarray.to_zarr does if the group does not already exist.

simply pass down the root group if the group arg is None

Yep! Generally we want to make this as similar as possible to xarray.Dataset.to_zarr().

I agree about kw-only args. We’re not too worried about backwards-compatibility at this point as the library is still new and evolving.