irods: iRODS storage tiering plugin does not create needed AVU when using irsync command or msi function
Feature/BUG
We implemented the tiered storage solution. For testing we create a file using iput. If we then move the file using irsync, msiDataObjRsync or msiCollRsync functions to an other federation it does not cause the file to move from TIER 0 to TIER 1 on the remote federation. NOTE: It also does not work locally. This is because there is no AVU being created. Can the appropiate pep function also be included in the storage tiering ruleset?
versions:
- iRODS:
$ rpm -qa | grep irods | sort
irods-database-plugin-postgres-4.2.12-1.x86_64
irods-devel-4.2.12-1.x86_64
...
irods-externals-avro1.9.0-0-1.0-1.x86_64
irods-externals-boost1.67.0-0-1.0-1.x86_64
irods-externals-catch22.3.0-0-1.0-1.x86_64
irods-externals-clang-runtime6.0-0-1.0-1.x86_64
irods-externals-cppzmq4.2.3-0-1.0-1.x86_64
irods-externals-fmt6.1.2-1-1.0-1.x86_64
irods-externals-json3.7.3-0-1.0-1.x86_64
irods-externals-libarchive3.3.2-1-1.0-1.x86_64
irods-externals-nanodbc2.13.0-1-1.0-1.x86_64
irods-externals-spdlog1.5.0-1-1.0-1.x86_64
irods-externals-zeromq4-14.1.6-0-1.0-1.x86_64
irods-icommands-4.2.12-1.x86_64
irods-rule-engine-plugin-python-4.2.12.0-1.x86_64
irods-rule-engine-plugin-unified-storage-tiering-4.2.12.0-1.x86_64
irods-runtime-4.2.12-1.x86_64
irods-server-4.2.12-1.x86_64
- OS:
$ cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
What did you try to do?
- create a file using iput.
- use irsync to put it in an othere federation
Expected behavior
expect the data/file to have an extra AVU at the remote source: irods::access_time
with a timestamp
the data/file is moved from TIER 0 to TIER 1
Observed behavior (including steps to reproduce, if applicable)
- create file:
iput eudatCreatePid_2.r test_data_org_igor_20230630_10.txt -R eudatCache
- copy file using irsync to remote federation:
$ irsync i:test_data_org_igor_20230630_10.txt i:/frank/home/robertv#igor/test_data_org_igor_20230630_11.txt -R eudatCache
- check file:
$ ils -l /frank/home/robertv#igor/test_data_org_igor_20230630_11.txt
robertv 0 eudatCache 1195 2023-06-30.10:37 & test_data_org_igor_20230630_11.txt
- check AVU:
$ imeta ls -d /frank/home/robertv#igor/test_data_org_igor_20230630_11.txt
AVUs defined for dataObj /frank/home/robertv#igor/test_data_org_igor_20230630_11.txt:
None
- check remote resource:
$ imeta ls -R eudatCache -z frank
AVUs defined for resource eudatCache:
attribute: irods::storage_tiering::group
value: eudat
units: 0
----
attribute: irods::storage_tiering::time
value: 120
units:
----
attribute: irods::storage_tiering::verification
value: filesystem
units:
Observed behavior (including steps to reproduce, if applicable)
It also does not work within the same zone:
$ irsync i:test_data_org_igor_20230630_10.txt i:test_data_org_igor_20230630_30.txt -R eudatCache
$ ils -l test_data_org_igor_20230630_30.txt
robertv 0 eudatCache 1195 2023-06-30.12:07 & test_data_org_igor_20230630_30.txt
$ imeta ls -d test_data_org_igor_20230630_30.txt
AVUs defined for dataObj /igor/home/robertv/test_data_org_igor_20230630_30.txt:
None
The local resource:
$ imeta ls -R eudatCache
AVUs defined for resource eudatCache:
attribute: irods::storage_tiering::group
value: eudat
units: 0
----
attribute: irods::storage_tiering::time
value: 120
units:
----
attribute: irods::storage_tiering::verification
value: filesystem
units:
Observed behavior (including steps to reproduce, if applicable)
We also have a ruleset were it is being used:
/opt/eudat/b2safe/rulebase/replication.re: msiCollRsync(*source,*destination,*dest_res,"IRODS_TO_IRODS",*rsyncStatus);
/opt/eudat/b2safe/rulebase/replication.re: msiDataObjRsync(*source,"IRODS_TO_IRODS",*dest_res,*destination,*rsyncStatus);
And that also does not work.
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 15 (1 by maintainers)
Thanks I implemented it with the
*err = errorcode(msi_atomic_apply_metadata_operations(args ...));
Thanks for the reply.
We will try an other approach. Create a rule which is executed at specific times which does following:
irods::access_time
After this the normal sequence will be done by the storage tiering plugin rule.