sysrepo: sr_rwrelock: Assertion `!rwlock->readers[0] && !rwlock->upgr && (rwlock->writer == cid)' failed
Hi, this is a netopeer2 crash (in sysrepo) that we experience in our large test suite. So far we had seen this erratically, at different locations. As we installed latest Netopeer2 2.4.1 / sysrepo 3.6.11 we saw further and clearer logs in the journal mentioning condition variables and, what’s more important, one of our stress tests started to fail rather deterministically.
I was therefore able to set a gdb and catch the event. I thought that I had forgot some unlock, but after having traced all of them with prints, I am in the idea that this condition happens when concurrent operations happen, even on different models, on the running datastore.
From netconf, we have this RPC being processed (printed via lyd_print_file in gdb):
netopeer2-server[83282]: <edit-config xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
netopeer2-server[83282]: <target>
netopeer2-server[83282]: <running/>
netopeer2-server[83282]: </target>
netopeer2-server[83282]: <config>
netopeer2-server[83282]: <hardware xmlns="urn:ietf:params:xml:ns:yang:ietf-hardware">
netopeer2-server[83282]: <component>
netopeer2-server[83282]: <name>O-RU</name>
netopeer2-server[83282]: <state>
netopeer2-server[83282]: <admin-state>locked</admin-state>
netopeer2-server[83282]: </state>
netopeer2-server[83282]: </component>
netopeer2-server[83282]: </hardware>
netopeer2-server[83282]: </config>
netopeer2-server[83282]: </edit-config>
concurrently, we have an internal event where we try to set a carrier active state up or down (depending on former admin lock/unlock) on the running datastore. On doing so, we set a lock to o-ran-uplane-conf model. It’s likely that the RPC is processed while the reactions on the previous iteration is being processed as well.
Why a setting on the running datastore? Because a setting on operational, for a param which is config:true, does not work. The other option is letting running and operational diverge by bypassing sysrepo through the backend, but I’m not sure what happens then on reads. AI suggested to set the running.
Ideally the order of execution would not matter because the RPC is on different object which causes further settings only later.
Thanks for your time
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 17 (4 by maintainers)
Commits related to this issue
- common BUGFIX avoid write-urge lock deadlock Test included. Refs #3642 — committed to sysrepo/sysrepo by michalvasko a year ago
Yes, confirmed
I also cherry picked the change and run again my test suite, now it is executing fine!
I’m closing this issue, looking forward for the next official release. Thanks a lot.
I’m guessing If you check
p/x mod->statefromsr_shmmod_modinfo_wrlock_downgrade()frame, and see that(mod->state & MOD_INFO_RLOCK_UPGR)will evaluate to true.My suspicion is that while your thread is downgrading a READ_UPGR to a READ lock, another thread was attempting a URGE lock (to lock the ds).
Now with the 9b828c7ec918c70fcc76a1f534e2c9f844986f98, a urge writer will let the upgr go first, so this situation cannot arise anymore, afaict.
I could only reproduce this after removing the fix
and inserting sleep in the code to get the timing right.
Regarding
operationalvs.running, you can have different values in each datastore, for configuration nodes as well and you always could. Just use standard means of settingoperationaldata and they will replace those inrunningwhen queryingoperational.