hvplot: hvplot is not using lazy loading for zarr-backed xarray dataset

ALL software version info

xr.__version__
'0.21.1'
hvplot.__version__
'0.7.3'
holoviews.__version__
'1.14.7'

Description of expected behavior and the observed behavior

When I visualize zarr-backed xarray datasets with hvplot, I expect the loading to be lazy. I don’t want to load the entire array into memory. I expect this to happen even without dask being involved.

Instead, I am noticing that hvplot is loading data eagerly.

Complete, minimal, self-contained example code that reproduces the issue

import xarray as xr
import hvplot.xarray

url = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/swot_adac/FESOM/surf/fma.zarr"
ds = xr.open_dataset(url, engine='zarr') # note that ds is not chunked but still uses lazy loading

# this will trigger loading the whole array into memory
ds.sst.hvplot(x='lon', y='lat', dynamic=True, rasterize=True)

Note that lazyness can be achieved by chunking the xarray dataset. But I don’t want to have to do that. Dask introduces additional latency that I would like to avoid here.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (12 by maintainers)

Most upvoted comments

I’m on xarray 2022.3.0 and this is sufficient to reproduce:

import xarray as xr

url = "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/swot_adac/FESOM/surf/fma.zarr"
ds = xr.open_dataset(url, engine='zarr') # note that ds is not chunked but still uses lazy loading
ds.chunks

There is a time dimension.

ds.isel(time=0)

will lazily grab a single time step, which is all that should be required for a single image plot.