iceoryx: mutex owner died -> POPO__CHUNK_LOCKING_ERROR

Hi, we just encountered a fatal crash in roudi (first time on b18411364d1cac72422c53a4b254fad0c9061204 since it came out), roudi logs below.

Maybe an iceoryx “client” process died prior, but no iceoryx logging in them or the syslog indicates this. And this should not crash roudi of course… -Do you maybe have any pointers as to why we encountered this and how it should be remedied ?

(We’re running 21 iceoryx client processes on the box, pinned and isolated, ~1500 publishers, and roudi on highest prio)

Thanks

Required information

Operating system: Ubuntu 22.04.2 LTS

Compiler version: Clang 17.0.5

Eclipse iceoryx version: b18411364d1cac72422c53a4b254fad0c9061204 (v2.0.x, 20231121)

Observed result or behaviour:

^[[0;90m2024-02-14 07:06:02.846 ^[[0;1;31;103m[Error]^[[m: /mnt/c/src/thirdparty/vcpkg/buildtrees/iceoryx/src/d0c9061204-f53d20e6d8.clean/iceoryx_hoofs/source/posix_wrapper/mutex.cpp:323 { expected<void, MutexLockError> iox::posix::mutex::lock() -> pthread_mutex_lock }  :::  [ 130 ]  Owner died
^[[0;90m2024-02-14 07:06:02.846 ^[[0;1;31;103m[Error]^[[m: The thread/process which owned the mutex died. The mutex is now in an inconsistent state and must be put into a consistent state again with Mutex::make_consistent()
^[[0;90m2024-02-14 07:06:02.846 ^[[0;1;97;41m[Fatal]^[[m: Locking of an inter-process mutex failed! This indicates that the application holding the lock was terminated or the resources were cleaned up by RouDi due to an unresponsive application.
^[[0;90m2024-02-14 07:06:02.846 ^[[0;1;31;103m[Error]^[[m: ICEORYX error! POPO__CHUNK_LOCKING_ERROR
libc++abi: terminating

Edit by elBoberido:

Related issues

• #325
• #2260