iris: Investigate segfaults with NetCDF4 `>1.6.0`
📰 Custom Issue
A bad interaction between Iris, Dask and NetCDF4 is causing segfaults when loading. This is apparently fixed if Dask is set to single-threaded mode - see https://github.com/Unidata/netcdf4-python/issues/1192#issuecomment-1267390240
This workaround removes a large part of the benefit of using Dask in Iris, so we need to find out if we can instead fix via changes to the Iris loading code.
Before the next release (v3.4), 1 of the below needs to happen:
- Permanent fix - change Iris’ loading code to avoid the segfaults
- Temporary workaround - pin NetCDF4
<=1.6.0 - Temporary workaround, with big performance regressions - set Dask to single-threaded within Iris
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 17 (10 by maintainers)
OK that PR got merged so iris should now be repo-patched to use only netCDF4 <1.6.1 (after nearly destroying your conda forge package by replacing all the deps with netCDF4 <1.6.1 instead of appending it to them 🤣 )
Sounds good. We can talk on #5095.
Martin, sure, I’ll take it for a spin a short bit! @bouweandela @zklaus may want to look at it as well, Cheers for working on this issue, mate 🍺
@valeriupredoi are any of you in ESMValTool able to test against a branch on my fork? #5095,
trexfeathers:netcdf_segssorry I forgot to get meself here, guys! No, not a problem at all - as @zklaus pointed out, moving to Python=3.11 gonna be a bit lengthy anyway since we’d have to wait for all our dependency ducks to get in line with the new Python. But in the long run (ie a few months) it’d be good to have it. Let me know if I can help you in any way BTW 🍺
On the urgency,
@valeriupredoi has recently said “pinning the package is not a long term solution”
@trexfeathers this was recently mentioned https://github.com/ESMValGroup/ESMValCore/issues/1776#issuecomment-1305549061
Ping @valeriupredoi 👍