h5py: [SWMR] can't open for append if file is being read by another process

Hi,

I want to do the following:

open a file for write (create a file and create a dataset in it)
close the file
open the file for read in an independent process that keeps it open
open the file in ‘a’ mode and extend the existing dataset

According to section 5.1 of the hdf5 SWMR docs this should be possible. Note that it explicitly states:

after step 1, the file can be opened by the writer and reader(s) in any order

But when I try (using, e.g. this code with h5py 2.6 and hdf5 v 1.10.0) it fails while trying to open in ‘a’ mode:

hdf5_version= 1.10.0
READER: Open file for read
Traceback (most recent call last):
  File "openbug2.py", line 48, in <module>
    open_for_append()
  File "openbug2.py", line 36, in open_for_append
    f = h5py.File(FNAME, 'a', libver='latest')
  File "/home/cpascual/.local/devenv/lib/python/site-packages/h5py-2.6.0-py2.7-linux-x86_64.egg/h5py/_hl/files.py", line 272, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
  File "/home/cpascual/.local/devenv/lib/python/site-packages/h5py-2.6.0-py2.7-linux-x86_64.egg/h5py/_hl/files.py", line 106, in make_fid
    fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/nfs/home/cpascual/src/h5py/h5py/_objects.c:2582)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/nfs/home/cpascual/src/h5py/h5py/_objects.c:2541)
  File "h5py/h5f.pyx", line 96, in h5py.h5f.create (/nfs/home/cpascual/src/h5py/h5py/h5f.c:1997)
IOError: Unable to create file (Unable to open file: name = 'foo.h5', errno = 17, error message = 'file exists', flags = 15, o_flags = c2)

Also, incidentally, note that the same code works if using hdf v 1.8.16 (which implies that existing code may break, even if no SWMR feature is used at all).

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 2
Comments: 27 (13 by maintainers)

Most upvoted comments

We hope to address the locking issue in1.10.1 later in the year. We will post the proposed solution on HDF forum. Stay tuned.

epourmal on Oct 8, 2016

Also, IMHO this is a regression which may break code that worked in 1.8.

Consider the following scenario:

A scientific data acquisition program writes on an hdf5 file. On each experiment, it opens the file, writes the acquired data and closes the file. Related experiments use different datagroups on the same file. Now, if a user is viewing the data from a previous experiment with a reader, the data acquisition will fail to re-open the file .

Note that the above happens even when no SWMR mode is used at all, while in v1.8, it was possible to work in the mentioned scenario (admittedly, risking to read inconsistent data, but at least the writer was not blocked and the reader could re-read later-on)

cpascual on Jun 2, 2016

Hi @aparamon ,thanks for pointing to the new docs. I agree in that they are now clear about the fact that what we intended is not currently allowed.

I just created an entry in the HDF Group forum with the feature request. Let’s see if someone gets interested:

https://forum.hdfgroup.org/t/allow-swmr-reader-to-be-set-before-swmr-writer/5244

cpascual on Jan 2, 2019

I think this defeats one of the main advantages of the SWMR feature (which is not requiring coordination between readers and writer).

cpascual on Jun 2, 2016