iris: Iris won't load netCDF files from the new CDS-beta
🐛 Bug Report
Iris won’t load netCDF files from the new CDS-beta. Someone has also reported this on a post about the updates to the CDS API here (https://forum.ecmwf.int/t/changes-to-grib-to-netcdf-converter-on-cds-beta-ads-beta/4322/19). I suspect the issue comes from the changes to their converter described there. However, I think iris not loading the files is an iris bug. If I change the offending line (traceback below) in iris from
total_bytes = cf_var.size * cf_var.dtype.itemsize
to
total_bytes = cf_var.size * np.dtype(cf_var.dtype).itemsize
Then the file loads fine. Simple fix but I thought it’s worth reporting in case this might lead to other issues elsewhere.
Running ncdump on the file shows up the difference too. The variables look like this. The string in front of the string variables wouldn’t have been there before. I don’t know if that is an issue or not.
double latitude(latitude) ;
latitude:_FillValue = NaN ;
string latitude:units = "degrees_north" ;
string latitude:standard_name = "latitude" ;
string latitude:long_name = "latitude" ;
string latitude:stored_direction = "decreasing" ;
How To Reproduce
Steps to reproduce the behaviour:
- Download a netCDF file from CDS-beta. I just ticked the first box for each value here (https://cds-beta.climate.copernicus.eu/datasets/reanalysis-era5-pressure-levels?tab=download), i.e. Divergence at 1hPa, 1st January 1940
- Try to open it with iris
Expected behaviour
The netCDF file should load. It works with xarray and CF python
Additional context
Click to expand this section...
AttributeError Traceback (most recent call last) Cell In[2], line 1 ----> 1 cubes = iris.load(“test_era5.nc”)
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/init.py:326, in load(uris, constraints, callback) 302 def load(uris, constraints=None, callback=None): 303 “”“Load any number of Cubes for each constraint. 304 305 For a full description of the arguments, please see the module (…) 324 325 “”” –> 326 return _load_collection(uris, constraints, callback).merged().cubes()
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/init.py:294, in _load_collection(uris, constraints, callback) 292 try: 293 cubes = _generate_cubes(uris, callback, constraints) –> 294 result = _CubeFilterCollection.from_cubes(cubes, constraints) 295 except EOFError as e: 296 raise iris.exceptions.TranslationError( 297 “The file appears empty or incomplete: {!r}”.format(str(e)) 298 )
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/cube.py:97, in _CubeFilterCollection.from_cubes(cubes, constraints) 95 pairs = [_CubeFilter(constraint) for constraint in constraints] 96 collection = _CubeFilterCollection(pairs) —> 97 for cube in cubes: 98 collection.add_cube(cube) 99 return collection
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/init.py:275, in _generate_cubes(uris, callback, constraints) 273 if scheme == “file”: 274 part_names = [x[1] for x in groups] –> 275 for cube in iris.io.load_files(part_names, callback, constraints): 276 yield cube 277 elif scheme in [“http”, “https”]:
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/io/init.py:219, in load_files(filenames, callback, constraints) 217 fnames = handler_map[handling_format_spec] 218 if handling_format_spec.constraint_aware_handler: –> 219 for cube in handling_format_spec.handler(fnames, callback, constraints): 220 yield cube 221 else:
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/netcdf/loader.py:641, in load_cubes(file_sources, callback, constraints) 638 if mesh is not None: 639 mesh_coords, mesh_dim = _build_mesh_coords(mesh, cf_var) –> 641 cube = _load_cube(engine, cf, cf_var, cf.filename) 643 # Attach the mesh (if present) to the cube. 644 for mesh_coord in mesh_coords:
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/netcdf/loader.py:326, in _load_cube(engine, cf, cf_var, filename) 324 these_settings = CHUNK_CONTROL.var_dim_chunksizes.get(cf_var.cf_name, {}) 325 with CHUNK_CONTROL.set(**these_settings): –> 326 return _load_cube_inner(engine, cf, cf_var, filename)
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/netcdf/loader.py:355, in _load_cube_inner(engine, cf, cf_var, filename) 350 _assert_case_specific_facts(engine, cf, cf_var.cf_group) 352 # Run the actions engine. 353 # This creates various cube elements and attaches them to the cube. 354 # It also records various other info on the engine, to be processed later. –> 355 engine.activate() 357 # Having run the rules, now add the “unused” attributes to each cf element. 358 def fix_attributes_all_elements(role_name):
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/_nc_load_rules/engine.py:95, in Engine.activate(self) 85 def activate(self): 86 “”“Run all the translation rules to produce a single output cube. 87 88 This implicitly references the output variable for this operation, (…) 93 94 “”” —> 95 run_actions(self)
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/_nc_load_rules/actions.py:575, in run_actions(engine) 573 auxcoord_facts = engine.fact_list(“auxiliary_coordinate”) 574 for auxcoord_fact in auxcoord_facts: –> 575 action_build_auxiliary_coordinate(engine, auxcoord_fact) 577 # Detect + process and special ‘ukmo’ attributes 578 # Run on every cube : they choose themselves whether to trigger. 579 action_ukmo_stash(engine)
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/_nc_load_rules/actions.py:92, in action_function.<locals>.inner(engine, *args, **kwargs) 89 @wraps(func) 90 def inner(engine, *args, **kwargs): 91 # Call the original rules-func —> 92 rule_name = func(engine, *args, **kwargs) 93 if rule_name is None: 94 # Work out the corresponding rule name, and log it. 95 # Note: an action returns a name string, which identifies it, 96 # but also may vary depending on whether it successfully 97 # triggered, and if so what it matched. 98 rule_name = _default_rulenamesfunc(func.name)
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/nc_load_rules/actions.py:410, in action_build_auxiliary_coordinate(engine, auxcoord_fact) 407 rule_name += f"{coord_type}" 409 cf_var = engine.cf_var.cf_group.auxiliary_coordinates[var_name] –> 410 hh.build_auxiliary_coordinate(engine, cf_var, coord_name=coord_name) 412 return rule_name
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/_nc_load_rules/helpers.py:1238, in build_auxiliary_coordinate(engine, cf_coord_var, coord_name, coord_system) 1236 points_data = cf_coord_var.cf_label_data(cf_var) 1237 else: -> 1238 points_data = _get_cf_var_data(cf_coord_var, engine.filename) 1240 # Get any coordinate bounds. 1241 cf_bounds_var, climatological = get_cf_bounds_var(cf_coord_var)
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/netcdf/loader.py:219, in _get_cf_var_data(cf_var, filename) 217 result = cf_var._data_array 218 else: –> 219 total_bytes = cf_var.size * cf_var.dtype.itemsize 220 if total_bytes < _LAZYVAR_MIN_BYTES: 221 # Don’t make a lazy array, as it will cost more memory AND more time to access. 222 # Instead fetch the data immediately, as a real array, and return that. 223 result = cf_var[:]
AttributeError: type object ‘str’ has no attribute ‘itemsize’
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 22 (21 by maintainers)
Filtering Warnings — Iris 3.11.1 documentation
Yea I can see from above you’ve already put alot of thought into this problem. I just thought it would be putting in my hat as a user that there’s a big need for this to be automated solution of some kind for the next release, so you get some feedback on the issue.
Although I kind-of disagreed about the “it’s only a coordinate” part, I’m increasingly agreeing with this that we should have an automatic solution. Big hint : This is already labelled for “3.12” !
I think this misses a possibly somewhat big point : the “expver” variable is really not a coordinate – except in xarray’s slightly odd way of looking at things : It is actually a data variable in its own right, which would make a cube, if we could only load it.
Since this is a pretty big flaw in iris for users (CDS data is only becoming more used) I think a hacky solution now (or this release) is better than a perfectly considered solution later (yes, I know there’s nothing more permament than a temperory solution, but not working on a huge class of external data is a big deal).
Given we can make sure we’re only guessing the chunk size for string coordinates, I don’t think it’s too much of a stretch to get all the data to determine chunk size. String coordinates aren’t large, and slowing down cases of loading lots of small files with many string coords is worth having CDS files just-work. It’s not something I think there’s a large demand for a warning for, given users already complain about warning fatigue with iris.
Monkey-patch “solution”
We tried a nasty quick fix like this :
It “works” at least to make loading possible. The resulting coord has an numpy object-array as its data, which might give trouble elsewhere (like saving it?), but presumably you could fix that yourself by deleting it or converting to something more standard.
We’ve had reports that the
ncksworkaround doesn’t work 100% of the time. The team are deployed on other priorities at the moment (Backlog · 🐔Iris v3.11 (github.com)), but here is a slightly more sophisticated workaround in the meantime: