pangeo-forge-recipes: Hanging during `store_chunk` loop

Just a quick note: I’m running a job to generate STAC items for a bunch of NetCDF files and it seems like the hanging might still be present with h5netcdf 0.12.0, at least for my workload 😦

Now I’m worried that xarray is creating similar references.

I think we already know roughly what the issue is, but here’s a potentially simpler reproducer, with just fsspec and h5netcdf. I suspect we can simplify this a bit by using h5py directly.

import fsspec
import h5netcdf
import h5py


def main():
    url = "https://pangeo.blob.core.windows.net/pangeo-public/hurs_day_ACCESS-CM2_historical_r1i1p1f1_gn_1950.nc"
    for i in range(100):
        with fsspec.open(url).open() as o:
            h5netcdf.core.File(o, mode="r", decode_vlen_strings=True)

        print("--------------------------------------", i)


if __name__ == "__main__":
    main()

It’s possible that xarray is be careful with circular references. At least my method for detecting them isn’t showing anything:

import fsspec
import planetary_computer
import xarray as xr
import pympler.garbagegraph
import pympler.refgraph
import gc

# read once to get all the imports cached
credential = planetary_computer.sas.get_token("nasagddp", "nex-gddp-cmip6").token
file = 'nex-gddp-cmip6/NEX/GDDP-CMIP6/ACCESS-CM2/ssp245/r1i1p1f1/hurs/hurs_day_ACCESS-CM2_ssp245_r1i1p1f1_gn_2044.nc'
url = f"https://nasagddp.blob.core.windows.net/{file}?{credential}"
f = fsspec.open(url).open()
ds = xr.open_dataset(f)

# read a second file to avoid the cache
file = 'nex-gddp-cmip6/NEX/GDDP-CMIP6/ACCESS-CM2/ssp245/r1i1p1f1/hurs/hurs_day_ACCESS-CM2_ssp245_r1i1p1f1_gn_2045.nc'
url = f"https://nasagddp.blob.core.windows.net/{file}?{credential}"
f = fsspec.open(url).open()

pympler.garbagegraph.start_debug_garbage()
ds = xr.open_dataset(f)

gb = pympler.garbagegraph.GarbageGraph()

gb.render("garbage2.pdf", format="pdf")

Which gives:

The cycle on the right is caused by a closure in h5netcdf that I missed.

The larger cycle on the left is from fsspec, with circular references between the filesystem, the file, cache, and the OpenFile object. With some surgery it possible to remove that cycle.

But I think this all underscores how fragile this whole I/O layering is. This is something that has to be rock solid. It feels like to do this (reading NetCDF files over the network) properly we need to be able to pass a URL to the C NetCDF library and have it do the proper range requests, etc. That also seems like a ton of work, so` I’m not really sure what to do. I guess we keep patching around things at the python layer.

h5netcdf 0.12.0 is out and hopefully fixes the problem. If anyone has a workflow to run that semi-reliably triggers the errors, it’d be great to test out before closing this issue (maybe @cisaacstern has one in mind?)

I spent another 30 minutes trying to get a pure h5py + fsspec reproducer, without success. For reference, I’m trying to reproduce the parts of h5netcdf.File.__init__ where things seem to hang:

import fsspec
import h5netcdf
import h5py
import gc
import logging


def main():
    url = "https://pangeo.blob.core.windows.net/pangeo-public/hurs_day_ACCESS-CM2_historical_r1i1p1f1_gn_1950.nc"
    for i in range(100):
        with fsspec.open(url).open() as o:
            # h5netcdf.core.File(o, mode="r", decode_vlen_strings=True)
            h5file = h5py.File(o)
            root = h5file["/"]
            # for key in ["hurs", "lat", "lon", "time"]:
            dim_variable = h5file["/time"]

            for ref, _ in dim_variable.attrs["REFERENCE_LIST"]:
                var = root[ref]
                for j, var_d in enumerate(var.dims):
                    print(j, "var_d", var_d)
                    var_d[0].name
                var[:]
        print("--------------------------------------", i)

AFAICT, that should be roughly equivalent. I might be making an error (most likely) but it’s possible this is meaningful. If this is actually an interaction between the garbage collector, fsspec, and h5py objects, then maybe we need to be making circular references that get cleaned up during the gc (i.e. h5netcdf is creating those circular references). I’ll open an issue on h5py now to see if we can get some additional help debugging.

edit: https://github.com/h5py/h5py/issues/2019

Ok I have it running. Mine hung on i=8.

I would propose copying this file to a fully public location to eliminate the token stuff. Then let’s open an issue in h5py. They will likely try to kick it back to fsspec, but at least we can try to move forward towards resolving.

Just noting in manual execution of https://github.com/pangeo-forge/staged-recipes/pull/97#issuecomment-990162893 today, this issue remains very much present. KeyboardInterrupting and restarting the loop from the hang location is the workaround.

Not really no. It’s more of a hope that #218 will close this.

for the present 50 occurrence test, it was always None

I tried to follow that path a little - glad it’s not the case, as I was really puzzled!

Thanks for continuing to push on this Charles. I think we need to get to the bottom fo the issue with a minimal reproducer.

In the meantime, I plan to work this week on implementing the “reference” reading approach.

Is there a way to completely disable the async layer of fsspec

Absolutely not - they call libraries that are implemented in pure-async.

Two other thing that may be worth trying for the sake of elimination:

• running with dask’s “sync” scheduler
• running without dask but using ordinary threads

OK, so explicitly its the mixture of xarray/h5py/fsspec/threads. If gc doesn’t make a difference, then I am once again at a loss.