pangeo-forge-recipes: StoreToZarr _invert_meshgrid() assertion error.

I’ve been running the beam-refactor branch following the example of netcdf_zarr_sequential.ipynb with a dataset with dimensions (time, height, lat, lon) and am running into an error at the following assertion.

https://github.com/pangeo-forge/pangeo-forge-recipes/blob/97a7b3c211263ada4d0d73770027af782f58e8af/pangeo_forge_recipes/rechunking.py#L126-L128

I’m wondering if the _invert_meshgrid() function does not generalize to more than two dimensions? Specifically, np.meshgrid() only acts on the first two dimensions, so the dimensions of “actual” and “expected” do not match. When I comment out this assertion, things seem to run.

See the full error message below:

Traceback

  Traceback (most recent call last):
  File "/homedirs/katibren/develop/pangeo-forge-recipes/pangeo_forge_recipes/rechunking.py", line 197, in combine_fragments
    starts = _invert_meshgrid(*starts_cube[::-1])[::-1]
  File "/homedirs/katibren/develop/pangeo-forge-recipes/pangeo_forge_recipes/rechunking.py", line 128, in _invert_meshgrid
    assert all(
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1418, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 625, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/transforms/core.py", line -1, in <lambda>
  File "/homedirs/katibren/develop/pangeo-forge-recipes/pangeo_forge_recipes/rechunking.py", line 200, in combine_fragments
    raise ValueError("Cannot combine fragments because they do not form a regular hypercube.")
ValueError: Cannot combine fragments because they do not form a regular hypercube.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/homedirs/katibren/develop/pangeo-forge-recipes/docs/pangeo_forge_recipes/tutorials/xarray_zarr/zarr_zarr_debug.py", line 105, in <module>
    with beam.Pipeline() as p:
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/pipeline.py", line 600, in __exit__
    self.result = self.run()
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/pipeline.py", line 577, in run
    return self.runner.run_pipeline(self, self._options)
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/runners/direct/direct_runner.py", line 131, in run_pipeline
    return runner.run_pipeline(pipeline, options)
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 202, in run_pipeline
    self._latest_run_result = self.run_via_runner_api(
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 224, in run_via_runner_api
    return self.run_stages(stage_context, stages)
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 455, in run_stages
    bundle_results = self._execute_bundle(
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 783, in _execute_bundle
    self._run_bundle(
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1012, in _run_bundle
    result, splits = bundle_manager.process_bundle(
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1348, in process_bundle
    result_future = self._worker_handler.control_conn.push(process_bundle_req)
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py", line 379, in push
    response = self.worker.do_instruction(request)
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py", line 624, in do_instruction
    return getattr(self, request_type)(
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py", line 662, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1062, in process_bundle
    input_op_by_transform_id[element.transform_id].process_encoded(
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 232, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1420, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1508, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1418, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 625, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "/homedirs/katibren/miniconda3/envs/pangeo-forge-beam/lib/python3.10/site-packages/apache_beam/transforms/core.py", line -1, in <lambda>
  File "/homedirs/katibren/develop/pangeo-forge-recipes/pangeo_forge_recipes/rechunking.py", line 200, in combine_fragments
    raise ValueError("Cannot combine fragments because they do not form a regular hypercube.")
ValueError: Cannot combine fragments because they do not form a regular hypercube. [while running 'Create|OpenWithXarray|PreprocessMyDataset|StoreToZarr/StoreToZarr/Rechunk/MapTuple(combine_fragments)']

About this issue

Original URL
State: closed
Created a year ago
Comments: 18 (17 by maintainers)

Most upvoted comments

Thanks for weighing in, all!

And apologies for the delay here.

I have blocked off tomorrow to work on this.

cisaacstern on Jan 24, 2024

Thanks, @cisaacstern! Although in my case, ignorance was to blame. I did get this working once I set target_chunk equal to a factor of len(time)! 🤕

thodson-usgs on Jan 25, 2024

Charles this is huge! This bug has been bugging us for almost a year!!!

🚀 🚀 🚀

rabernat on Mar 15, 2024

Thanks for raising this issue. I agree it may be related to #517, but difficult to say for sure without a minimal reproducible example.

@kbren, could you post a self-contained example of the recipe you’re working on, which when executed raises this error?

Thanks!

cisaacstern on May 1, 2023

Credit to @jbusecke and @ta-hill for major contributions towards the fix!

cisaacstern on Mar 15, 2024

Thanks all for your patience. To the best of my understanding, this is now fixed by #708 and released in 0.10.6. Please re-open this issue or open a new one if you find otherwise! Thanks for everyone’s patience and collaboration on this!

cisaacstern on Mar 15, 2024

I think I’m hitting this one also with my ssebop recipe, which fails if time chunk !=1.

thodson-usgs on Jan 24, 2024

Let me know if I can help in any way here. This has quite a high priority on my end.

jbusecke on Nov 30, 2023

@cisaacstern thanks for the heads up! I’m just getting back from vacation. I’ll check to see if this is resolved, but it might take me a few days to get to it.

kbren on Jul 17, 2023