iris: Derived coordinates are not loaded anymore?
📰 Custom Issue
Hey guys, one more for me today, I am using this file and loading it simple:
import iris
cs = iris.load("/home/valeriu/ESMValCore/tests/integration/cmor/_fixes/test_data/common_cl_ap.nc")
for c in cs:
print(c)
with iris=3 I am not seeing the derived coordinates no more:
- with
iris=2.4:
b_bnds / (1) (atmosphere_hybrid_sigma_pressure_coordinate: 2; -- : 2)
Dimension coordinates:
atmosphere_hybrid_sigma_pressure_coordinate x -
Auxiliary coordinates:
ap x -
b x -
Derived coordinates:
air_pressure x -
ap_bnds / (1) (atmosphere_hybrid_sigma_pressure_coordinate: 2; -- : 2)
Dimension coordinates:
atmosphere_hybrid_sigma_pressure_coordinate x -
Auxiliary coordinates:
ap x -
b x -
Derived coordinates:
air_pressure x -
cloud_area_fraction_in_atmosphere_layer (time: 1; atmosphere_hybrid_sigma_pressure_coordinate: 2; latitude: 3; longitude: 4)
Dimension coordinates:
time x - - -
atmosphere_hybrid_sigma_pressure_coordinate - x - -
latitude - - x -
longitude - - - x
Auxiliary coordinates:
surface_air_pressure x - x x
ap - x - -
b - x - -
Derived coordinates:
air_pressure x x x x
surface_air_pressure / (Pa) (time: 1; latitude: 3; longitude: 4)
Dimension coordinates:
time x - -
latitude - x -
longitude - - x
Attributes:
additional_attribute: xyz
- with
iris=3.0:
b_bnds / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 2; -- : 2)
Dimension coordinates:
atmosphere_hybrid_sigma_pressure_coordinate x -
Auxiliary coordinates:
ap x -
b x -
ap_bnds / (unknown) (atmosphere_hybrid_sigma_pressure_coordinate: 2; -- : 2)
Dimension coordinates:
atmosphere_hybrid_sigma_pressure_coordinate x -
Auxiliary coordinates:
ap x -
b x -
cloud_area_fraction_in_atmosphere_layer (time: 1; atmosphere_hybrid_sigma_pressure_coordinate: 2; latitude: 3; longitude: 4)
Dimension coordinates:
time x - - -
atmosphere_hybrid_sigma_pressure_coordinate - x - -
latitude - - x -
longitude - - - x
Auxiliary coordinates:
surface_air_pressure x - x x
ap - x - -
b - x - -
surface_air_pressure / (Pa) (time: 1; latitude: 3; longitude: 4)
Dimension coordinates:
time x - -
latitude - x -
longitude - - x
Attributes:
additional_attribute: xyz
where did the Derived coordinates go?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 64 (45 by maintainers)
@bouweandela @valeriupredoi @schlunma @jvegasbsc
iris3.0.1 has already been tagged, the docs have already been built on readthedocs, and I’ve just kicked of an update on theconda-forgeiris-feedstock… almost there 😉I’ll ping the ESMValTool WhatsApp thread when
conda-forgehas pushed the builtiris3.0.1 package to the Anaconda Cloud channel - then you can install and test. Good luck! 🤞LOL what a muppet… hang tight…
Ha! Found one:
/CMIP6/CMIP/BCC/BCC-ESM1/historical/r1i1p1f1/Amon/cl/gn/v20181217/cl_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc@schlunma seems like a reasonable work around, but it’s really causing you guys quite a lot of extra work to jump through hoops.
I’ve got a fix that I’m testing just now that would gracefully promote any formula terms variable that must be dimensionless to be dimensionless if it has
unitsofunknown- it’s doing this within the aux factory itself.Also, this behavioural change applies to 6 aux factories - so for me, it makes more sense to fix this in
irisrather than force this upon all users.I’m going to roll with this fix regardlessly, as my expectation is that this will affect many users.
Question: do you have any other ESMValTool netCDF files that also show this failure that I can test my fix with?
@valeriupredoi Okay, I’m dropping everything today to service this - I can totally appreciate the situation you’re in, so I’m going to push an
iris3.0.1 release today.Hopefully, I can turn this around quickly for you, as I know you’re working to a tight deadline.
I’ll ping you here @valeriupredoi with my progress on this to keep you in the loop 👍
@valeriupredoi Okay, so this took a wee bit of digging, but the mystery is thankfully solved 🕵️
Firstly, I can recreate your issue in
iris3.0.0, and I can also confirm thatiris2.4.0 does indeed realise derived coordinates for your example dataset 👍This change in behaviour between
iris3.0.0 and 2.4.0 directly relates to PR #3795, where theunitsof anAuxCoodorDimCoordthat is not explicitly specified defaults tounknownrather then1(dimensionless).We thought about this change in behaviour long and hard, as the treatment of default
unitswas inconsistent acrossirisfor different objects, and the CF Metadata standard did not add any clarity (no surprises there, indeed it was inconsistent), but in the end we unified the behaviour for consistency sake. That is, erring on the side that it correct and safer to assignunitsasunknownin the case when they are not specified, whereas it is potentially incorrect and not safe to assume default dimensionless units instead.This change in behaviour is mentioned in the
iris3.0.0 release notes in the Incompatible Changes section, fifth bullet point.Where this is stinging you is for the missing
unitsfor thesigmavariable associated with theatmosphere_hybrid_sigma_pressure_coordinatei.e., givenformula_terms = "ap: ap_bnds b: b_bnds ps: ps",sigmain this case is the NetCDFbvariable, and this is defined within the example file simply asdouble b(lev) ;As the
bvariable has nounitsassociated with itiris3.0.0 assigns the defaultunitsofunknown(which they are). However, during the creation of the auxiliary factory that creates the derived coordinate it enforces the contract that thesigmavariable must bedimensionless, which in this case it’s not, as it’sunknown. Note that, enforcing thatsigmais dimensionless has not changed from 2.4.0 to 3.0.0.This failure for
sigmaunitsresults inirisraising awarningand not creating the associatedHybridPressureFactoryauxiliary factory i.e., during loading you see the following warning:Also, note the example given in the CF Metadata Conventions, where their
sigmavariable has explicitunits = "1".So in order to fix this, my advise is to add the correct
unitsto yoursigmavariable in the file. In addition to this, you could also add the missingboundsattribute for thesigma(b) anddelta(ap) variables e.g.,Hope this helps 😄
Installed
iris3.0.1 and tested… seems to work for me 😉Thanks for the amazingly quick bugfix and release! 🥳
legend @bjlittle cheers very much! 🍺
Cheers @bjlittle, I’m happy with it 👍 Thanks for your quick help!!
Unless you tell me otherwise, I’m cracking on with 3.0.1…
@valeriupredoi thanks, nuke the repo dude 👍
that’s terrific! Thanks very much @bjlittle - a very nice example of cross-package collaboration when you have cool devvs on both sides, remind me to buy you a 🍺 when all this mess is over (the pandemic not the ESGF data issues, the latter one won’t be over soon) 😁
The
UKESMmodel should have analtitudederived coordinate, that’s weird!EDIT: You need to use
fname2instead offnamein the beginning 😄yay! welcome to the absolute nightmare on ESGF street @bjlittle 🤣
Sweet, this looks good too…
I’ve got one file, the other is almost there… so glad we did this as the file
cl_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412contains the alternativeformula_terms = "p0: p0 a: a b: b ps: ps"for aatmosphere_hybrid_sigma_pressure_coordinate, whereap(delta) is formed by performingp0 * a.In the example file
ahas nounitsandp0hasunitsofPa(which does and must matchps), however thep0 * aresults in adeltawithunitsofunknownand causes the same failure, but in a different way.I’ve located the fix in
iris/fileformats/netcdf.pybefore it then creates the aux factory, as it’s too late for the patch in the aux factory to fix this particular nuance.Re-testing now…
Blimey it’s a sloooooow connection… 😴
@bjlittle You can just download the files directly from ESGF:
I’ve put two of those files on JASMIN at
/home/users/valeriu/badDatawhich is open to all JASMIN users, unfortunately the local server that I use to get into JASMIN is read-only today (as it happens, when you need something done), and I can’tscpthem anywhere w/o first going through it - any chance any of yous can grab them straight from my JASMIN location?You can check
clof all CMIP6 models, I guess there’s more of them. I have a meeting now, can help you guys later! 👍Thanks V.!
I found another one with hybrid height:
MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/cl/gn/v20190406/cl_Amon_UKESM1-0-LL_historical_r1i1p1f2_gn_185001-189912.nc.told you they’re like ants 😆 That’s on the ESGF node (at DKRZ I assume, since Manu works on that one), let me try grab it off CEDA-JASMIN and put it somewhere where you can download it
No worries, that’s not a problem. Thanks for checking, much appreciated 👍
This has turned out to be such a simple change, I’m pretty confident that similar files with no
unitsattributes on NetCDF variables that should have explicitunits = 1(dimensionless) will work (pride before a fall 🤕)massive 👏 @bjlittle - unfortunately both @schlunma and me couldn’t find anymore of these messed files, but there is a good probability there may be out there, given that these are like ants - once you see one they’re bound to be more
cirrus-ci is certainly happy with the fix i.e., no side-effects that our testing coverage can pick up… that’s encouraging (or we have a hole in our test coverage 😆)
I’m just going to check our unit tests and ensure there is test coverage for at least this change… then we should be good to release.
@bjlittle Thanks, that sounds like a great solution!
@bjlittle hang on, I’m rerunning the tests, try find more files, if @schlunma doesn’t beat me to it 😃
you’re legend @bjlittle cheers muchly! 🍺
@valeriupredoi Okay, no worries dude. Leave it with me and I’ll get back to you asap… just need to speak to the team about this 👍
@valeriupredoi Done a quick hack to see, if in principle, this promotion of
unitsofunknownto dimensionless (1) within auxiliary factories works…seems possible