flox: memory error with large expected_groups

Hi, I’m testing out flox (with dask) as replacement for scala based zonal stats on global rasters (30m resolution) and getting promising results in performance with cleaner and much smaller code! However I’m running into this memory issue that I wanted to see if has cleaner solution than what I’m doing now.

Here is the simple code for calculating tree cover loss area at three political boundary levels, grouped by a total of six dask array layers with expected_groups sizes of (23, 5, 7, 248, 854, 86).

tcl_by_year = xarray_reduce(
    areas.band_data,
    tcl_data,
    drivers_data,
    tcd_thresholds_data,
    gadm_adm0_data,
    gadm_adm1_data,
    gadm_adm2_data,
    func='sum',
    expected_groups=(tcl_years, drivers_cats, tcd_threshold_levels, gadm_adm0_ids, gadm_adm1_ids, gadm_adm2_ids),
).compute()

That runs into this error:

MemoryError: Unable to allocate 109 GiB for an array with shape (14662360160,) and data type int64

I’m getting around this by chunking the group-by layer with the highest unique labels (854) and doing the above in a dask delayed function over the chunks and concatenating the results.

from dask import delayed
chunk_size = 200
tasks = [
    delayed(reduce_chunk)(gadm_adm2_ids[i:i+chunk_size])
    for i in range(0, len(gadm_adm2_ids), chunk_size)
]

results = dask.compute(*tasks)

combined = xr.concat(results, dim="gadm_adm2")

chunked = final.chunk({'tcl_year':-1, 'drivers':-1, 'tcd_threshold': -1, 'gadm_adm0':1, 'gadm_adm2':-1, 'gadm_adm1':-1})

chunked.to_zarr("s3://**/gadm_results.zarr", mode="w")
Image

This works but I may not be persisting some of these layers correctly for use by the expected_group chunks and runs slower than expected. Is there a more efficient and elegant way to handle this situation? It’d be great for example if this MultiIndex can be built dynamically in the workers with the available groups. Thanks.

Environment: Flox: 0.10.1 Dask: 2025.2.0 Numpy: 2.2.3 Xarray: 2025.1.2

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22

Commits related to this issue

Most upvoted comments

That sounds good, @dcherian. I’m persisting the rasters on read which I think explains the memory spike at the start – not flox related which uses much less memory and is a lot faster with the simple test case you put together. I’ll be able to investigate more in a couple of week and will share any details here.

I’m going to close this but would be interested in diagnosing that memory use in the beginning. I don’t think it’s flox related though

The whole run including reading the geotiffs (in our roadmap to migrate to zarr for analysis) located in the same aws region took about 24 minutes (I’ll have to disaggregate this) and cost about $9. Had to ran it on the 64GB nodes as some step (aligning?) is taking a lot of memory but will need to spend more time on the dask dashboard to diagnose that issue!

Image

Ah, so close! that was the very last step. Should be fixed in #440

I also fixed another case where the RangeIndex was getting realized in to memory, so you should see faster graph construction too I think.

The sparse ValueError above may have been because of the mismatch between the data and the expected_groups types, not having noticed the type change (user error:).

Weird, that shouldn’t affect things but seems like you don’t see it any more.

I’m testing this with 50 64GB workers.

You should be able to use a lot less memory now!

I actually tried reindex=False but I get ValueError: Please provide expected_groups if not grouping by a numpy array. error. The same xarray_reduce call as above with that kwarg added and removing expected_groups. Am I using it wrong?

Forgot to respond, yes you’ll need to provide expected_groups when grouping by dask arrays AND using Xarray. Are you able to try out #430?