h5py: Custom HDF5 plugin can't find libhdf5.so when using h5py

In preparation for a future satellite mission, the provider of the data has implemented a compression method based on charls as a plugin to HDF5, FCIDECOMP, last filter on this page: https://support.hdfgroup.org/services/filters.html

The building and using of this filter works as expected with h5dump:

❯ h5dump -d effective_radiance sample.nc
HDF5 "sample.nc" {
DATASET "effective_radiance" {
   DATATYPE  H5T_STD_I16LE
   DATASPACE  SIMPLE { ( 60, 30 ) / ( 60, 30 ) }
   DATA {
   (0,0): 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
   (0,19): 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29,
...

(here is the test file for completeness: sample.zip, 1.3kB)

However, trying to achieve the same thing from h5py leads to an error:

>>> import h5py
>>> h5f = h5py.File('sample.nc', mode='r')
>>> h5f['effective_radiance'][:]

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/a001673/miniconda3/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 651, in __getitem__
    return self._fast_reader.read(args)
  File "h5py/_reader.pyx", line 243, in h5py._reader.Reader.read
OSError: Can't read data (can't dlopen:/home/a001673/miniconda3/envs/fcidecomp3/hdf5/lib/plugin/libH5Zjpegls.so: undefined symbol: H5Pmodify_filter)

The HDF_PLUGIN_PATH and LD_LIBRARY_PATH are setup correctly to:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib
export HDF5_PLUGIN_PATH=:$CONDA_PREFIX/hdf5/lib/plugin

again, as h5dump has no issue reading the file. (The filter FCIDECOMP was built by me as a conda package)

Running nm -D on libH5Zjpegls.so lists of course H5Pmodify_filter as undefined, but it is present in libhdf5.so which itself is located on the defined LD_LIBRARY_PATH.

So, I don’t know how to go further in the debugging here, I would need some assistance 😃

1. Have custom filters been tested and reported to work with h5py ?
2. Is there some LD_LIBRARY_PATH magic happening in the h5py code, and in this case where can I find it ?
3. Any other idea of things I could try ?

To assist reproducing bugs, please include the following:

• Operating System (e.g. Windows 10, MacOS 10.11, Ubuntu 16.04.2 LTS, CentOS 7) RHEL8
• Python version (e.g. 2.7, 3.5) 3.7.4
• Where Python was acquired (e.g. system Python on MacOS or Linux, Anaconda on Windows) Miniconda
• h5py version (e.g. 2.6) Latest from conda-forge (2.10.0), then github’s master branch, fetched one hour ago
• HDF5 version (e.g. 1.8.17) 1.10.6_nompi, from conda-forge

Summary of the h5py configuration
---------------------------------

h5py    2.10.0
HDF5    1.10.6
Python  3.7.4 (default, Aug 13 2019, 20:35:49) 
[GCC 7.3.0]
sys.platform    linux
sys.maxsize     9223372036854775807
numpy   1.18.1