irods: iRODS storage tiering plugin does not create needed AVU when using irsync command or msi function

Feature/BUG

We implemented the tiered storage solution. For testing we create a file using iput. If we then move the file using irsync, msiDataObjRsync or msiCollRsync functions to an other federation it does not cause the file to move from TIER 0 to TIER 1 on the remote federation. NOTE: It also does not work locally. This is because there is no AVU being created. Can the appropiate pep function also be included in the storage tiering ruleset?

versions:

  • iRODS:
$ rpm -qa | grep irods | sort
irods-database-plugin-postgres-4.2.12-1.x86_64
irods-devel-4.2.12-1.x86_64
...
irods-externals-avro1.9.0-0-1.0-1.x86_64
irods-externals-boost1.67.0-0-1.0-1.x86_64
irods-externals-catch22.3.0-0-1.0-1.x86_64
irods-externals-clang-runtime6.0-0-1.0-1.x86_64
irods-externals-cppzmq4.2.3-0-1.0-1.x86_64
irods-externals-fmt6.1.2-1-1.0-1.x86_64
irods-externals-json3.7.3-0-1.0-1.x86_64
irods-externals-libarchive3.3.2-1-1.0-1.x86_64
irods-externals-nanodbc2.13.0-1-1.0-1.x86_64
irods-externals-spdlog1.5.0-1-1.0-1.x86_64
irods-externals-zeromq4-14.1.6-0-1.0-1.x86_64
irods-icommands-4.2.12-1.x86_64
irods-rule-engine-plugin-python-4.2.12.0-1.x86_64
irods-rule-engine-plugin-unified-storage-tiering-4.2.12.0-1.x86_64
irods-runtime-4.2.12-1.x86_64
irods-server-4.2.12-1.x86_64
  • OS:
$ cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

What did you try to do?

  • create a file using iput.
  • use irsync to put it in an othere federation

Expected behavior

expect the data/file to have an extra AVU at the remote source: irods::access_time with a timestamp the data/file is moved from TIER 0 to TIER 1

Observed behavior (including steps to reproduce, if applicable)

  • create file:
iput eudatCreatePid_2.r test_data_org_igor_20230630_10.txt -R eudatCache
  • copy file using irsync to remote federation:
$ irsync i:test_data_org_igor_20230630_10.txt i:/frank/home/robertv#igor/test_data_org_igor_20230630_11.txt -R eudatCache
  • check file:
$ ils -l /frank/home/robertv#igor/test_data_org_igor_20230630_11.txt
  robertv           0 eudatCache         1195 2023-06-30.10:37 & test_data_org_igor_20230630_11.txt
  • check AVU:
$ imeta ls -d /frank/home/robertv#igor/test_data_org_igor_20230630_11.txt
AVUs defined for dataObj /frank/home/robertv#igor/test_data_org_igor_20230630_11.txt:
None
  • check remote resource:
$ imeta ls -R eudatCache -z frank
AVUs defined for resource eudatCache:
attribute: irods::storage_tiering::group
value: eudat
units: 0
----
attribute: irods::storage_tiering::time
value: 120
units:
----
attribute: irods::storage_tiering::verification
value: filesystem
units:

Observed behavior (including steps to reproduce, if applicable)

It also does not work within the same zone:

$ irsync i:test_data_org_igor_20230630_10.txt i:test_data_org_igor_20230630_30.txt -R eudatCache

$ ils -l test_data_org_igor_20230630_30.txt
  robertv           0 eudatCache         1195 2023-06-30.12:07 & test_data_org_igor_20230630_30.txt

$ imeta ls -d test_data_org_igor_20230630_30.txt
AVUs defined for dataObj /igor/home/robertv/test_data_org_igor_20230630_30.txt:
None

The local resource:

$ imeta ls -R eudatCache
AVUs defined for resource eudatCache:
attribute: irods::storage_tiering::group
value: eudat
units: 0
----
attribute: irods::storage_tiering::time
value: 120
units:
----
attribute: irods::storage_tiering::verification
value: filesystem
units:

Observed behavior (including steps to reproduce, if applicable)

We also have a ruleset were it is being used:

/opt/eudat/b2safe/rulebase/replication.re:                msiCollRsync(*source,*destination,*dest_res,"IRODS_TO_IRODS",*rsyncStatus);
/opt/eudat/b2safe/rulebase/replication.re:                msiDataObjRsync(*source,"IRODS_TO_IRODS",*dest_res,*destination,*rsyncStatus);

And that also does not work.

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 15 (1 by maintainers)

Most upvoted comments

Thanks I implemented it with the *err = errorcode(msi_atomic_apply_metadata_operations(args ...));

Thanks for the reply.

We will try an other approach. Create a rule which is executed at specific times which does following:

  • find all files in TIER 0
  • check if the appropiate AVU has been set irods::access_time
  • If the AVU has not been set, set it.

After this the normal sequence will be done by the storage tiering plugin rule.