cfgrib: If opening with xarray open_mfdataset and parallel=True it will fail unless you have previously opened it with parallel=False

I confirm this bug report with a different dataset and different error messages.

With parallel=False open_mfdataset always work:

>>> import cfgrib
>>> import xarray as xr
>>> print(xr.__version__, cfgrib.__version__)
0.13.0 0.9.7.4.dev0
>>> ds = xr.open_mfdataset('step*.grib', engine='cfgrib', concat_dim=['step'], combine='nested', parallel=False)
>>> ds
<xarray.Dataset>
Dimensions:     (latitude: 1801, longitude: 3600, step: 3)
Coordinates:
    time        datetime64[ns] 2019-04-01
    number      int64 0
    surface     int64 0
  * latitude    (latitude) float64 90.0 89.9 89.8 89.7 ... -89.8 -89.9 -90.0
  * longitude   (longitude) float64 0.0 0.1 0.2 0.3 ... 359.6 359.7 359.8 359.9
  * step        (step) timedelta64[ns] 01:00:00 02:00:00 03:00:00
    valid_time  (step) datetime64[ns] 2019-04-01T01:00:00 ... 2019-04-01T03:00:00
Data variables:
    t2m         (step, latitude, longitude) float32 dask.array<chunksize=(1, 1801, 3600), meta=np.ndarray>
Attributes:
    GRIB_edition:            1
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2019-11-11T19:19:05 GRIB to CDM+CF via cfgrib-0....

Restarting the kernel and running with parallel=True always crashes python inside ecCodes but it returns a few different error messages. I observed at leat:

ECCODES ERROR   :  Unable to find boot.def. Context path=/Users/amici/.conda/envs/ECM/share/eccodes/definitions

Possible causes:
- The software is not correctly installed
- The environment variable ECCODES_DEFINITION_PATH is defined but incorrect

ecCodes assertion failed: `0' in /usr/local/miniconda/conda-bld/eccodes_1566402639979/work/src/grib_context.c:205

ECCODES ERROR   :  grib_handle_create: cannot create handle, no definitions found
ecCodes assertion failed: `h' in /usr/local/miniconda/conda-bld/eccodes_1566402639979/work/src/grib_query.c:458
ECCODES ERROR   :  grib_parser: syntax error at line 34 of /Users/amici/.conda/envs/ECM/share/eccodes/definitions/boot.def
ECCODES ERROR   :  ecCodes Version: 2.13.1

and

ECCODES ERROR   :  ecCodes Version: 2.13.1
ecCodes Version:       2.13.1
Definition files path: /Users/amici/.conda/envs/ECM/share/eccodes/definitions
ECCODES ERROR   :  grib_parser_include: Could not resolve 'ECCODES_USE_' (included in /Users/amici/.conda/envs/ECM/share/eccodes/definitions/boot.def)
ecCodes assertion failed: `0' in /usr/local/miniconda/conda-bld/eccodes_1566402639979/work/src/grib_context.c:205

It looks like a locking/threading problem, @shahramn do you have any hint?

Hi @guidocioni , I’m not sure if you’re on macos or Linux, but we’ve managed to update the conda version. Could you do the following: conda search eccodes -c conda-forge and if you see a 2.18.0 version with _1 at the end, install that version please. It takes the conda servers a little while to update their indexes, but it’s appeared now at least on macos.

Sounds like you’re on the right path. A few years ago, when cfgrib was still a baby 😃, I was getting an error while trying to read compressed grib files as the recipe for eccodes on conda was not including the compression library because of a license issue. So in the end the problem was on the eccodes side on conda.

Awesome. Thanks for looking into it. Not all heroes wear capes 😃

On 08.10.2020 15:01, shahramn wrote:

Looks like the conda recipe does NOT enable the thread safety flags. I will look into this

– You are receiving this because you commented. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/ecmwf/cfgrib/issues/110#issuecomment-705551762 [2] https://github.com/notifications/unsubscribe-auth/ALQB7NON65KU5AOROSXQJJDSJWZ3NANCNFSM4JKA7GMQ

Looks like the conda recipe does NOT enable the thread safety flags. I will look into this

The ecCodes library has to be built with thread safety enabled See https://confluence.ecmwf.int/display/UDOC/Is+ecCodes+thread-safe+-+ecCodes+FAQ

I can confirm this is still here on xarray 0.16.1 and cfgrib 0.9.8.4. For now I’m using parallel = False but it takes about 3 times longer than with parallel = True. The problem is that when opening the files for the first time with parallel = True eccodes throw an error to cfrgib which is unable to write idx files. The error which you then see in python is due to empty idx files.