irods: Storage tier migration does not recover from the destination resource filling temporarily

Bug Report

iRODS Version, OS and Version

4.2.7, Linux, Ubuntu 18.04.6 LTS

What did you try to do?

Allow the destination resource of storage-tiering to fill, then make space on the filled resource partition by deleting some data objects that had already migrated there.

This is the state of the resources before the experiment:

$ iquest "select RESC_NAME, sum(DATA_SIZE) where COLL_NAME like '/tempZone/home/srpipe/%'"
RESC_NAME = resource3
DATA_SIZE = 1111111110000
------------------------------------------------------------

(resource1 and resource2 also exist, but are empty, everything having already been migrated to resource3)

Expected behavior

On creating more free space, the tiering operations to restart the next time a relevant rule fired. iquest to show an accurate DATA_SIZE occupancy at some point for a query like:

`iquest “select RESC_NAME, sum(DATA_SIZE) where COLL_NAME like ‘/tempZone/home/myuser/somepath%’”

Observed behavior (including steps to reproduce, if applicable)

Data objects migrate successfully until the disk is filled. Any further data data objects selected for tier migration fail with an error (below) as expected. However, the catalog does not appear to get updated with respect to DATA_SIZE.

If the successfully migrated data objects are cleanly removed with irm (no errors server-side), it remains impossible to migrate further data objects, even though disk space is now available. The errors this yields are the same as if the resource were still full. Moreover, running iquest as above shows that the catalog records the occupancy as the “start” value i.e. the almost-full state before adding a few tens more GB to fill the resource to capacity. It’s even impossible to migrate empty files at this point.

$ iquest "select RESC_NAME, sum(DATA_SIZE) where COLL_NAME like '/tempZone/home/srpipe/%'"
RESC_NAME = resource3
DATA_SIZE = 1111111110000
------------------------------------------------------------

Example rodsLog error raised when an object fails to migrate. The data object in question is 1 GB and the destination resource is 92% full, with 94 GB of free space.

Jan 31 10:34:13 pid:10523 NOTICE: found 4 objects for resc [root] with query [SELECT DATA_NAME, COLL_NAME, USER_NAME, DATA_REPL_NUM WHERE META_DATA_ATTR_NAME = 'tier:single-copy' AND META_DATA_ATTR_VALUE = '1' AND DATA_RESC_HIER like 'root;replicate%'] type [0]
Jan 31 10:34:42 pid:10580 remote addresses: 127.0.0.1, 172.27.208.232 ERROR: [-]	/irods/server/api/src/rsDataObjRepl.cpp:855:int dataObjOpenForRepl(rsComm_t *, dataObjInp_t *, dataObjInfo_t *, const char *, dataObjInfo_t *, int) :  status [NO_NEXT_RESC_FOUND]  errno [] -- message [failed in irods::resolve_resource_hierarchy for [/tempZone/home/srpipe/test_1gb_000]]
	[-]	/irods/server/core/src/irods_resource_redirect.cpp:293:irods::error irods::resolve_hier_for_create(rsComm_t *, irods::file_object_ptr, const char *, dataObjInp_t *, std::string &) :  status [NO_NEXT_RESC_FOUND]  errno [] -- message [failed in call to redirect host [irods-test-dev-ies1]  hier []]
		[-]	/irods/server/core/src/irods_resource_redirect.cpp:76:irods::error irods::request_vote_for_file_object(rsComm_t *, const std::string &, const std::string &, file_object_ptr, std::string &, float &) :  status [NO_NEXT_RESC_FOUND]  errno [] -- message [failed in call to redirect host [irods-test-dev-ies1]  hier []]
			[-]	/irods/plugins/resources/random/librandom.cpp:832:irods::error random_file_resolve_hierarchy(irods::plugin_context &, const std::string *, const std::string *, irods::hierarchy_parser *, float *) :  status [NO_NEXT_RESC_FOUND]  errno [] -- message [Failed getting next valid child.]
				[-]	/irods/plugins/resources/random/librandom.cpp:760:irods::error get_next_valid_child_resource(irods::plugin_context &, const std::string *, const std::string *, irods::hierarchy_parser *, float *) :  status [NO_NEXT_RESC_FOUND]  errno [] -- message [No valid child found.]

Jan 31 10:34:42 pid:10580 remote addresses: 127.0.0.1, 172.27.208.232 ERROR: [-]	/irods/server/api/src/rsDataObjRepl.cpp:855:int dataObjOpenForRepl(rsComm_t *, dataObjInp_t *, dataObjInfo_t *, const char *, dataObjInfo_t *, int) :  status [NO_NEXT_RESC_FOUND]  errno [] -- message [failed in irods::resolve_resource_hierarchy for [/tempZone/home/srpipe/test_1gb_000]]
	[-]	/irods/server/core/src/irods_resource_redirect.cpp:293:irods::error irods::resolve_hier_for_create(rsComm_t *, irods::file_object_ptr, const char *, dataObjInp_t *, std::string &) :  status [NO_NEXT_RESC_FOUND]  errno [] -- message [failed in call to redirect host [irods-test-dev-ies1]  hier []]
		[-]	/irods/server/core/src/irods_resource_redirect.cpp:76:irods::error irods::request_vote_for_file_object(rsComm_t *, const std::string &, const std::string &, file_object_ptr, std::string &, float &) :  status [NO_NEXT_RESC_FOUND]  errno [] -- message [failed in call to redirect host [irods-test-dev-ies1]  hier []]
			[-]	/irods/plugins/resources/random/librandom.cpp:832:irods::error random_file_resolve_hierarchy(irods::plugin_context &, const std::string *, const std::string *, irods::hierarchy_parser *, float *) :  status [NO_NEXT_RESC_FOUND]  errno [] -- message [Failed getting next valid child.]
				[-]	/irods/plugins/resources/random/librandom.cpp:760:irods::error get_next_valid_child_resource(irods::plugin_context &, const std::string *, const std::string *, irods::hierarchy_parser *, float *) :  status [NO_NEXT_RESC_FOUND]  errno [] -- message [No valid child found.]

Jan 31 10:34:42 pid:10580 NOTICE: rsDataObjRepl - Failed to replicate data object.
Level 0: failed to migrate [/tempZone/home/srpipe/test_1gb_000] to [noReplRoot]
Level 1: iRODS Exception:
    file: /irods_plugin/libirods_rule_engine_plugin-data_replication.cpp
    function: void (anonymous namespace)::replicate_object_to_resource(rsComm_t *, const std::string &, const std::string &, const std::string &, const std::string &)
    line: 38
    code: -1805000 (NO_NEXT_RESC_FOUND)
    message:
        failed to migrate [/tempZone/home/srpipe/test_1gb_000] to [noReplRoot]
        : [-]	/irods/server/re/include/irods_re_plugin.hpp:325:irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo *, irods::rule_execution_manager_pack::DONT_AUDIT_RULE>::call(std::string, std::string, OP, As &&...) [T = std::__1::tuple<>, C = RuleExecInfo *, Audit = irods::rule_execution_manager_pack::DONT_AUDIT_RULE, OP = std::__1::function<irods::error (const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack &&)>, As = <const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack>] :  status [NO_NEXT_RESC
Jan 31 10:34:42 pid:10580 remote addresses: 127.0.0.1, 172.27.208.232 ERROR: rsExecRuleExpression : -1805000, [-]	/irods_plugin/libirods_rule_engine_plugin-storage_tiering.cpp:531:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) :  status [NO_NEXT_RESC_FOUND]  errno [] -- message [iRODS Exception:
    file: /irods_plugin/storage_tiering_utilities.cpp
    function: void irods::invoke_policy(ruleExecInfo_t *, const std::string &, std::list<boost::any>)
    line: 88
    code: -1805000 (NO_NEXT_RESC_FOUND)
    message:
        iRODS Exception:
    file: /irods_plugin/storage_tiering_utilities.cpp
    function: void irods::invoke_policy(ruleExecInfo_t *, const std::string &, std::list<boost::any>)
    line: 88
    code: -1805000 (NO_NEXT_RESC_FOUND)
    message:
        iRODS Exception:
    file: /irods_plugin/libirods_rule_engine_plugin-data_replication.cpp
    function: void (anonymous namespace)::replicate_object_to_resource(rsComm_t *, const std::string &, const std::string &, const std::string &, const std::string &)
    line: 38
    code: -1805000 (NO_NEXT_RESC_FOUND)
    message:
        failed to migrate [/tempZone/home/srpipe/test_1gb_000] to [noReplRoot]
        : [-]	/irods/server/re/include/irods_re_plugin.hpp:325:irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo *, irods::rule_execution_manager_pack::DONT_AUDIT_RULE>::call(std::string, std::string, OP, As &&...) [T = std::__1::tuple<>, C = RuleExecInfo *, Audit = irods::rule_execution_manager_pack::DONT_AUDIT_RULE, OP = std::__1::function<irods::error (const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack &&)>, As = <const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack>] :  status [NO_NEXT_RESC_FOUND]  errno [] -- message []
stack trace:
--------------

Dumping stack trace
<0>	Offset: 0x65  	Address: 0x7f562e4d09e5	irods::exception::exception(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<1>	Offset: 0x43  	Address: 0x7f562e4d0b63	irods::exception::exception(long, boost::basic_format<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<2>	Offset: 0x751 	Address: 0x7f562c75f701	exec_rule(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)
<3>	Offset: 0x2e  	Address: 0x7f562c76db8e	std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback&&)
<4>	Offset: 0xf5  	Address: 0x7f56319e9615	irods::error irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::tuple<>&, irods::unpack&&, irods::callback)
<5>	Offset: 0x68  	Address: 0x7f56319e94a8	std::__1::__function::__func<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&), std::__1::allocator<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)
<6>	Offset: 0xfe  	Address: 0x7f56319e9f0e	irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, irods::re_pack_inp<std::__1::tuple<> >&&&, irods::unpack&&)::'lambda'()::operator()() const
<7>	Offset: 0x9f  	Address: 0x7f56319e921f	irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, irods::re_pack_inp<std::__1::tuple<> >&&&, irods::unpack&&)
<8>	Offset: 0xe3  	Address: 0x7f56319e8e73	irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const
<9>	Offset: 0xec  	Address: 0x7f56319e8bac	irods::error irods::control<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), std::__1::tuple<>, irods::unpack>(std::__1::list<irods::re_pack_inp<std::__1::tuple<> >, std::__1::allocator<irods::re_pack_inp<std::__1::tuple<> > > >&, irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)
<10>	Offset: 0xe7  	Address: 0x7f562c521c37	irods::invoke_policy(RuleExecInfo*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >)
<11>	Offset: 0x323 	Address: 0x7f562c512383	exec_rule(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)
<12>	Offset: 0x2e  	Address: 0x7f562c520eae	std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback&&)
<13>	Offset: 0xf5  	Address: 0x7f56319e9615	irods::error irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::tuple<>&, irods::unpack&&, irods::callback)
<14>	Offset: 0x68  	Address: 0x7f56319e94a8	std::__1::__function::__func<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&), std::__1::allocator<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)
<15>	Offset: 0xfe  	Address: 0x7f56319e9f0e	irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, irods::re_pack_inp<std::__1::tuple<> >&&&, irods::unpack&&)::'lambda'()::operator()() const
<16>	Offset: 0x9f  	Address: 0x7f56319e921f	irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, irods::re_pack_inp<std::__1::tuple<> >&&&, irods::unpack&&)
<17>	Offset: 0xe3  	Address: 0x7f56319e8e73	irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const
<18>	Offset: 0xec  	Address: 0x7f56319e8bac	irods::error irods::control<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), std::__1::tuple<>, irods::unpack>(std::__1::list<irods::re_pack_inp<std::__1::tuple<> >, std::__1::allocator<irods::re_pack_inp<std::__1::tuple<> > > >&, irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)
<19>	Offset: 0xe7  	Address: 0x7f562ceb0f67	irods::invoke_policy(RuleExecInfo*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >)
<20>	Offset: 0x9ba 	Address: 0x7f562ce7b14a	exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<21>	Offset: 0x2f  	Address: 0x7f562ce9363f	std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&)
<22>	Offset: 0xf0  	Address: 0x7f56315e4640	irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<23>	Offset: 0x354 	Address: 0x7f56315e44d4	irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*)
<24>	Offset: 0x25e 	Address: 0x7f56315e3eae	rsExecRuleExpression(rsComm_t*, exec_rule_expression_t*)
<25>	Offset: 0x44  	Address: 0x7f563183c534	irods::api_call_adaptor<exec_rule_expression_t*>::operator()(irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)
<26>	Offset: 0x13  	Address: 0x7f563183c4b3	std::__1::__function::__func<irods::api_call_adaptor<exec_rule_expression_t*>, std::__1::allocator<irods::api_call_adaptor<exec_rule_expression_t*> >, irods::error (irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)>::operator()(irods::plugin_context&, rsComm_t*&&, exec_rule_expression_t*&&)
<27>	Offset: 0x931 	Address: 0x7f5631731c31	int irods::api_entry::call_handler<exec_rule_expression_t*>(rsComm_t*, exec_rule_expression_t*)
<28>	Offset: 0x75b 	Address: 0x7f56318be83b	rsApiHandler(rsComm_t*, int, BytesBuf*, BytesBuf*)
<29>	Offset: 0x8f2 	Address: 0x7f56318c05b2	readAndProcClientMsg(rsComm_t*, int)
<30>	Offset: 0x9c7 	Address: 0x7f56318b40a7	agentMain(rsComm_t*)
<31>	Offset: 0x1cc3	Address: 0x7f56318b2a23	runIrodsAgentFactory(sockaddr_un)
<32>	Offset:       	Address: 0x4215bd	/usr/sbin/irodsServer(main+0x4dd) [0x4215bd]
<33>	Offset:       	Address: 0x7f562dbafbf7	/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f562dbafbf7]
<34>	Offset:       	Address: 0x421009	/usr/sbin/irodsServer(_start+0x29) [0x421009]



        : [-]	/irods/server/re/include/irods_re_plugin.hpp:325:irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo *, irods::rule_execution_manager_pack::DONT_AUDIT_RULE>::call(std::string, std::string, OP, As &&...) [T = std::__1::tuple<>, C = RuleExecInfo *, Audit = irods::rule_execution_manager_pack::DONT_AUDIT_RULE, OP = std::__1::function<irods::error (const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack &&)>, As = <const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack>] :  status [NO_NEXT_RESC_FOUND]  errno [] -- message []
stack trace:
--------------

Dumping stack trace
<0>	Offset: 0x65  	Address: 0x7f562e4d09e5	irods::exception::exception(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<1>	Offset: 0x58d 	Address: 0x7f562c5220dd	irods::invoke_policy(RuleExecInfo*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >)
<2>	Offset: 0x323 	Address: 0x7f562c512383	exec_rule(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)
<3>	Offset: 0x2e  	Address: 0x7f562c520eae	std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >&, irods::callback&&)
<4>	Offset: 0xf5  	Address: 0x7f56319e9615	irods::error irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::tuple<>&, irods::unpack&&, irods::callback)
<5>	Offset: 0x68  	Address: 0x7f56319e94a8	std::__1::__function::__func<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&), std::__1::allocator<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)
<6>	Offset: 0xfe  	Address: 0x7f56319e9f0e	irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, irods::re_pack_inp<std::__1::tuple<> >&&&, irods::unpack&&)::'lambda'()::operator()() const
<7>	Offset: 0x9f  	Address: 0x7f56319e921f	irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, irods::unpack&&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, irods::re_pack_inp<std::__1::tuple<> >&&&, irods::unpack&&)
<8>	Offset: 0xe3  	Address: 0x7f56319e8e73	irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&) const
<9>	Offset: 0xec  	Address: 0x7f56319e8bac	irods::error irods::control<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), std::__1::tuple<>, irods::unpack>(std::__1::list<irods::re_pack_inp<std::__1::tuple<> >, std::__1::allocator<irods::re_pack_inp<std::__1::tuple<> > > >&, irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<irods::unpack>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::unpack&&)
<10>	Offset: 0xe7  	Address: 0x7f562ceb0f67	irods::invoke_policy(RuleExecInfo*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >)
<11>	Offset: 0x9ba 	Address: 0x7f562ce7b14a	exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<12>	Offset: 0x2f  	Address: 0x7f562ce9363f	std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&)
<13>	Offset: 0xf0  	Address: 0x7f56315e4640	irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<14>	Offset: 0x354 	Address: 0x7f56315e44d4	irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*)
<15>	Offset: 0x25e 	Address: 0x7f56315e3eae	rsExecRuleExpression(rsComm_t*, exec_rule_expression_t*)
<16>	Offset: 0x44  	Address: 0x7f563183c534	irods::api_call_adaptor<exec_rule_expression_t*>::operator()(irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)
<17>	Offset: 0x13  	Address: 0x7f563183c4b3	std::__1::__function::__func<irods::api_call_adaptor<exec_rule_expression_t*>, std::__1::allocator<irods::api_call_adaptor<exec_rule_expression_t*> >, irods::error (irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)>::operator()(irods::plugin_context&, rsComm_t*&&, exec_rule_expression_t*&&)
<18>	Offset: 0x931 	Address: 0x7f5631731c31	int irods::api_entry::call_handler<exec_rule_expression_t*>(rsComm_t*, exec_rule_expression_t*)
<19>	Offset: 0x75b 	Address: 0x7f56318be83b	rsApiHandler(rsComm_t*, int, BytesBuf*, BytesBuf*)
<20>	Offset: 0x8f2 	Address: 0x7f56318c05b2	readAndProcClientMsg(rsComm_t*, int)
<21>	Offset: 0x9c7 	Address: 0x7f56318b40a7	agentMain(rsComm_t*)
<22>	Offset: 0x1cc3	Address: 0x7f56318b2a23	runIrodsAgentFactory(sockaddr_un)
<23>	Offset:       	Address: 0x4215bd	/usr/sbin/irodsServer(main+0x4dd) [0x4215bd]
<24>	Offset:       	Address: 0x7f562dbafbf7	/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f562dbafbf7]
<25>	Offset:       	Address: 0x421009	/usr/sbin/irodsServer(_start+0x29) [0x421009]



stack trace:
--------------

Dumping stack trace
<0>	Offset: 0x65  	Address: 0x7f562e4d09e5	irods::exception::exception(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<1>	Offset: 0x58d 	Address: 0x7f562ceb140d	irods::invoke_policy(RuleExecInfo*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::list<boost::any, std::__1::allocator<boost::any> >)
<2>	Offset: 0x9ba 	Address: 0x7f562ce7b14a	exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<3>	Offset: 0x2f  	Address: 0x7f562ce9363f	std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&)
<4>	Offset: 0xf0  	Address: 0x7f56315e4640	irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<5>	Offset: 0x354 	Address: 0x7f56315e44d4	irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*)
<6>	Offset: 0x25e 	Address: 0x7f56315e3eae	rsExecRuleExpression(rsComm_t*, exec_rule_expression_t*)
<7>	Offset: 0x44  	Address: 0x7f563183c534	irods::api_call_adaptor<exec_rule_expression_t*>::operator()(irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)
<8>	Offset: 0x13  	Address: 0x7f563183c4b3	std::__1::__function::__func<irods::api_call_adaptor<exec_rule_expression_t*>, std::__1::allocator<irods::api_call_adaptor<exec_rule_expression_t*> >, irods::error (irods::plugin_context&, rsComm_t*, exec_rule_expression_t*)>::operator()(irods::plugin_context&, rsComm_t*&&, exec_rule_expression_t*&&)
<9>	Offset: 0x931 	Address: 0x7f5631731c31	int irods::api_entry::call_handler<exec_rule_expression_t*>(rsComm_t*, exec_rule_expression_t*)
<10>	Offset: 0x75b 	Address: 0x7f56318be83b	rsApiHandler(rsComm_t*, int, BytesBuf*, BytesBuf*)
<11>	Offset: 0x8f2 	Address: 0x7f56318c05b2	readAndProcClientMsg(rsComm_t*, int)
<12>	Offset: 0x9c7 	Address: 0x7f56318b40a7	agentMain(rsComm_t*)
<13>	Offset: 0x1cc3	Address: 0x7f56318b2a23	runIrodsAgentFactory(sockaddr_un)
<14>	Offset:       	Address: 0x4215bd	/usr/sbin/irodsServer(main+0x4dd) [0x4215bd]
<15>	Offset:       	Address: 0x7f562dbafbf7	/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f562dbafbf7]
<16>	Offset:       	Address: 0x421009	/usr/sbin/irodsServer(_start+0x29) [0x421009]

About this issue

Original URL
State: open
Created 2 years ago
Comments: 58 (28 by maintainers)

Most upvoted comments

You are correct about the serial vs parallel operations you mention. I’ll do some more parallel testing.

However, I think you’re just seeing the soft edges of the asynchronous/mark-and-sweep system we’ve constructed.

Each create operation will trigger the freespace check prior to the create (which compares the value in r_resc_main.free_space for the target resource with the minimum_free_space_for_create_in_bytes value in r_resc_main.resc_context).

Updating the value in r_resc_main.free_space happens ‘independently’ by either a PEP or manually via iadmin modresc.

With concurrent iput operations firing, each will do the check, succeed with the create, and then the post-pep will fire and update the catalog with the newly calculated freespace. Multiple iput operations would get in the door before the freespace value was updated and drop below the minimum value.

The mitigation here is to treat the minimum as a soft floor, and increase the minimum accordingly if you know your system will be under concurrent write loads.

I left a couple files in the units: irods::storage_tiering::migration_scheduled state until their migration rules timed out (since I was not using FOR EVER). At this point, they were effectively ‘stranded/beached’ and needed manual intervention. Simply removing that unit allowed them to be picked up again by the next sweep (given enough free space, of course).

So, an additional simple tool could be built that can scan for these stranded data objects and… heal them or show them to the administrator…

iquest "select COLL_NAME, DATA_NAME where META_DATA_ATTR_UNITS = 'irods::storage_tiering::migration_scheduled'"

trel on Feb 8, 2022

pep_resource_rmdir_post

pep_api_file_truncate_post

pep_api_file_unlink_post

Planning on going with this;

pep_resource_rmdir_post(*leaf_resource) {
    msi_update_unixfilesystem_resource_free_space(*leaf_resource);
}
pep_api_file_truncate_post(*leaf_resource) {
    msi_update_unixfilesystem_resource_free_space(*leaf_resource);
}
pep_api_file_unlink_post(*leaf_resource) {
    msi_update_unixfilesystem_resource_free_space(*leaf_resource);
}

kript on Feb 3, 2022