xarray: `rename_vars` followed by `swap_dims` and `merge` causes swapped dim to reappear

What happened?

I wanted to rename a dimension coordinate for two datasets before merging: ds = ds.rename_vars(y="z").swap_dims(y="z"), and the same for the second data set. After merging the datasets, the merged result has the dimension “y” in addition to “z”.

Swapping the order of rename_vars and swap_dims before merging works in that “y” does not reappear, but then “z” is listed as a non-dimension coordinate.

Doing rename_vars followed by swap_dims /after/ merging gives the result I wanted, but if I merge again, the same issue occurs.

My current solution is to only rename dimension coordinates before saving to netCDF.

What did you expect to happen?

Merging two datasets with the same coordinates and dimensions (but different data variables) should result in a single dataset with all of the data variables from the two datasets and exactly the same coordinates and dimensions.

Minimal Complete Verifiable Example

import numpy as np
import xarray as xr
from xarray.core.utils import Frozen


A = np.arange(4).reshape((2, 2))
B = np.arange(4).reshape((2, 2)) + 4
ds1 = xr.Dataset({"A": (["x", "y"], A), "B": (["x", "y"], B)}, coords={"x": ("x", [1, 2]), "y": ("y", [1, 2])})
ds2 = xr.Dataset({"C": (["x", "y"], A), "D": (["x", "y"], B)}, coords={"x": ("x", [1, 2]), "y": ("y", [1, 2])})

assert ds1.dims == Frozen({"x": 2, "y": 2})
assert ds2.dims == Frozen({"x": 2, "y": 2})

ds1_swap = ds1.rename_vars(y="z").swap_dims(y="z")
ds2_swap = ds2.rename_vars(y="z").swap_dims(y="z")

assert ds1_swap.dims == Frozen({"x": 2, "z": 2})
assert ds2_swap.dims == Frozen({"x": 2, "z": 2})

# merging makes the dimension "y" reappear (I would expect this assertion to fail):
assert xr.merge([ds1_swap, ds2_swap]).dims == Frozen({"x": 2, "z": 2, "y": 2})

# renaming and swapping after the merge causes issues later:
ds12 = xr.merge([ds1, ds2]).rename_vars(y="z").swap_dims(y="z")
ds3 = xr.Dataset({"E": (["x", "z"], A), "F": (["x", "z"], B)}, coords={"x": ("x", [1, 2]), "z": ("z", [1, 2])})

# ds12 and ds3 have the same dimensions:
assert ds12.dims == Frozen({"x": 2, "z": 2})
assert ds3.dims == Frozen({"x": 2, "z": 2})

# but merging brings back "y"
ds123 = xr.merge([ds12, ds3])
assert ds123.dims == Frozen({"x": 2, "z": 2, "y": 2})

# as do other operations:
ds12_as = ds12.assign_coords(x=(ds12.x + 1))
assert ds12_as.sizes == Frozen({"x": 2, "z": 2, "y": 2})

MVCE confirmation

  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
  • Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

The MVCE works in all venvs I’ve tried including:

INSTALLED VERSIONS

commit: None python: 3.10.13 (main, Nov 10 2023, 15:02:19) [GCC 11.4.0] python-bits: 64 OS: Linux OS-release: 6.5.0-14-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: (‘en_GB’, ‘UTF-8’) libhdf5: 1.12.2 libnetcdf: 4.9.3-development

xarray: 2023.11.0 pandas: 1.5.3 numpy: 1.26.2 scipy: 1.11.4 netCDF4: 1.6.5 pydap: None h5netcdf: 1.3.0 h5py: 3.10.0 Nio: None zarr: None cftime: 1.6.3 nc_time_axis: 1.4.1 iris: None bottleneck: None dask: 2023.12.0 distributed: None matplotlib: 3.8.2 cartopy: 0.22.0 seaborn: 0.13.0 numbagg: None fsspec: 2023.12.1 cupy: None pint: None sparse: 0.15.1 flox: None numpy_groupies: None setuptools: 69.0.2 pip: 23.3.1 conda: None pytest: 7.4.3 mypy: None IPython: 8.18.1 sphinx: None /home/brendan/Documents/inversions/.pymc_venv/lib/python3.10/site-packages/_distutils_hack/init.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn(“Setuptools is replacing distutils.”)

INSTALLED VERSIONS

commit: None python: 3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0] python-bits: 64 OS: Linux OS-release: 3.10.0-1160.81.1.el7.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: (‘en_GB’, ‘UTF-8’) libhdf5: None libnetcdf: None

xarray: 2024.1.0 pandas: 2.2.0 numpy: 1.26.3 scipy: None netCDF4: None pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: None nc_time_axis: None iris: None bottleneck: None dask: None distributed: None matplotlib: None cartopy: None seaborn: None numbagg: None fsspec: None cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 69.0.3 pip: 23.3.2 conda: None pytest: None mypy: None IPython: 8.18.1 sphinx: None

About this issue

  • Original URL
  • State: open
  • Created 5 months ago
  • Comments: 15 (6 by maintainers)

Most upvoted comments

if these two:

ds = xr.Dataset(coords={"x": ("x", [0, 1]), "y": ("x", [-1, 1])})

ds.swap_dims({"x": "y"})
ds.rename_dims({"x": "y"}).set_xindex("y").drop_indexes(["x"])

do the same thing, could we just make the former use the latter as implementation (obviously adapted to allow renaming multiple dimensions at the same time)? I don’t think we’d even need a deprecation cycle for that.