sysrepo: sr_rwrelock: Assertion `!rwlock->readers[0] && !rwlock->upgr && (rwlock->writer == cid)' failed

Hi, this is a netopeer2 crash (in sysrepo) that we experience in our large test suite. So far we had seen this erratically, at different locations. As we installed latest Netopeer2 2.4.1 / sysrepo 3.6.11 we saw further and clearer logs in the journal mentioning condition variables and, what’s more important, one of our stress tests started to fail rather deterministically.

I was therefore able to set a gdb and catch the event. I thought that I had forgot some unlock, but after having traced all of them with prints, I am in the idea that this condition happens when concurrent operations happen, even on different models, on the running datastore.

From netconf, we have this RPC being processed (printed via lyd_print_file in gdb):

netopeer2-server[83282]: <edit-config xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
netopeer2-server[83282]:   <target>
netopeer2-server[83282]:     <running/>
netopeer2-server[83282]:   </target>
netopeer2-server[83282]:   <config>
netopeer2-server[83282]:     <hardware xmlns="urn:ietf:params:xml:ns:yang:ietf-hardware">
netopeer2-server[83282]:       <component>
netopeer2-server[83282]:         <name>O-RU</name>
netopeer2-server[83282]:         <state>
netopeer2-server[83282]:           <admin-state>locked</admin-state>
netopeer2-server[83282]:         </state>
netopeer2-server[83282]:       </component>
netopeer2-server[83282]:     </hardware>
netopeer2-server[83282]:   </config>
netopeer2-server[83282]: </edit-config>

concurrently, we have an internal event where we try to set a carrier active state up or down (depending on former admin lock/unlock) on the running datastore. On doing so, we set a lock to o-ran-uplane-conf model. It’s likely that the RPC is processed while the reactions on the previous iteration is being processed as well.

Why a setting on the running datastore? Because a setting on operational, for a param which is config:true, does not work. The other option is letting running and operational diverge by bypassing sysrepo through the backend, but I’m not sure what happens then on reads. AI suggested to set the running.

Ideally the order of execution would not matter because the RPC is on different object which causes further settings only later.

Thanks for your time

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 17 (4 by maintainers)

Commits related to this issue

Most upvoted comments

Yes, confirmed

(gdb) p /x mod->state
$3 = 0x126
(gdb) p /x (mod->state & 0x20)
$4 = 0x20

I also cherry picked the change and run again my test suite, now it is executing fine!

I’m closing this issue, looking forward for the next official release. Thanks a lot.

I’m guessing If you check p/x mod->state from sr_shmmod_modinfo_wrlock_downgrade() frame, and see that (mod->state & MOD_INFO_RLOCK_UPGR) will evaluate to true.

My suspicion is that while your thread is downgrading a READ_UPGR to a READ lock, another thread was attempting a URGE lock (to lock the ds).

Now with the 9b828c7ec918c70fcc76a1f534e2c9f844986f98, a urge writer will let the upgr go first, so this situation cannot arise anymore, afaict.

test_apply_changes: /home/irfan/new/sysrepo/src/common.c:2826: sr_rwrelock: Assertion `!rwlock->readers[0] && !rwlock->upgr && (rwlock->writer == cid)' failed.

Thread 2 "test_apply_chan" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff7a2a6c0 (LWP 2585958)]
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007ffff7c49f4f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x00007ffff7bfafb2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff7be5472 in __GI_abort () at ./stdlib/abort.c:79
#4  0x00007ffff7be5395 in __assert_fail_base (fmt=0x7ffff7d59a90 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x7ffff7f69118 "!rwlock->readers[0] && !rwlock->upgr && (rwlock->writer == cid)",
    file=file@entry=0x7ffff7f685f8 "/home/irfan/new/sysrepo/src/common.c", line=line@entry=2826, function=function@entry=0x7ffff7f69b70 <__PRETTY_FUNCTION__.37> "sr_rwrelock")
    at ./assert/assert.c:94
#5  0x00007ffff7bf3ec2 in __GI___assert_fail (assertion=0x7ffff7f69118 "!rwlock->readers[0] && !rwlock->upgr && (rwlock->writer == cid)",
    file=0x7ffff7f685f8 "/home/irfan/new/sysrepo/src/common.c", line=2826, function=0x7ffff7f69b70 <__PRETTY_FUNCTION__.37> "sr_rwrelock") at ./assert/assert.c:103
#6  0x00007ffff7f03d86 in sr_rwrelock (rwlock=0x7ffff7fbb938, timeout_ms=5000, mode=SR_LOCK_READ, cid=1, func=0x7ffff7f6ffb0 <__func__.8> "sr_shmmod_lock", cb=0x0, cb_data=0x0)
    at /home/irfan/new/sysrepo/src/common.c:2826
#7  0x00007ffff7f3d7d1 in sr_shmmod_lock (shm_lock=0x7ffff7fbb938, timeout_ms=5000, mode=SR_LOCK_READ, ds_timeout_ms=0, cid=1, sid=3, relock=1, mod_name=0x5555556baa60 "when2")
    at /home/irfan/new/sysrepo/src/shm_mod.c:1327
#8  0x00007ffff7f3df95 in sr_shmmod_modinfo_wrlock_downgrade (mod_info=0x7ffff7a29a60, sid=3, timeout_ms=5000) at /home/irfan/new/sysrepo/src/shm_mod.c:1558
#9  0x00007ffff7ef2fed in sr_changes_notify_store (mod_info=0x7ffff7a29a60, session=0x7ffff000ffe0, shmmod_session_del=0, timeout_ms=5000, has_change_sub_lock=SR_LOCK_NONE,
    err_info2=0x7ffff7a29aa8) at /home/irfan/new/sysrepo/src/sysrepo.c:4426
#10 0x00007ffff7ef371c in sr_apply_changes (session=0x7ffff000ffe0, timeout_ms=5000) at /home/irfan/new/sysrepo/src/sysrepo.c:4622
#11 0x000055555557619e in make_changes_when2 (arg=0x555555580e70) at /home/irfan/new/sysrepo/tests/test_apply_changes.c:8208
#12 0x00007ffff7c481f5 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007ffff7cc889c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb) f 6
#6  0x00007ffff7f03d86 in sr_rwrelock (rwlock=0x7ffff7fbb938, timeout_ms=5000, mode=SR_LOCK_READ, cid=1, func=0x7ffff7f6ffb0 <__func__.8> "sr_shmmod_lock", cb=0x0, cb_data=0x0)
    at /home/irfan/new/sysrepo/src/common.c:2826
2826        assert(!rwlock->readers[0] && !rwlock->upgr && (rwlock->writer == cid));
(gdb) p *rwlock
$1 = {mutex = {__data = {__lock = 0, __count = 1, __owner = 0, __nusers = 0, __kind = 146, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
    __size = "\000\000\000\000\001", '\000' <repeats 11 times>, "\222", '\000' <repeats 22 times>, __align = 4294967296}, cond = {futex = 95}, readers = {1,
    0 <repeats 15 times>}, read_count = "\001", '\000' <repeats 14 times>, upgr = 1, writer = 1}
(gdb)

I could only reproduce this after removing the fix

diff --git a/src/common.c b/src/common.c
index a20fbfa1..5a538446 100644
--- a/src/common.c
+++ b/src/common.c
@@ -2505,7 +2505,7 @@ sr_sub_rwlock(sr_rwlock_t *rwlock, struct timespec *timeout_abs, sr_lock_mode_t
         ret = 0;
         wr_urged = 0;
         while (!ret && (rwlock->readers[0] || (rwlock->writer && !wr_urged))) {
-            if (!rwlock->writer && !rwlock->upgr) {
+            if (!rwlock->writer) {
                 /* urge waiting for write lock, but let upgradeable read lock go first (to avoid dead-lock, it will not
                  * release its write lock) */
                 rwlock->writer = cid;

and inserting sleep in the code to get the timing right.

Regarding operational vs. running, you can have different values in each datastore, for configuration nodes as well and you always could. Just use standard means of setting operational data and they will replace those in running when querying operational.