mediasoup: Mediasoup worker crash (libwebrtc uint64_t/int64_t conversion related)
Bug Report
Mediasoup worker dies unexpectedly
Your environment
- Operating system: Alpine Linux v3.11
- Node version: v12.14.0
- npm version: 6.13.4
- gcc/clang version: gcc 9.2.0
- mediasoup version: v3.4.5
- mediasoup-client version: 3.3.12
Issue description
Occasionally we observe a worker dying on our production servers. This is what we see in the logs:
2020-01-06T14:47:05.196Z mediasoup:ERROR:worker[pid:22] RTC::DtlsTransport::CheckStatus() | OpenSSL error [desc:'SSL status: SSL_ERROR_SSL', error:'error:140E0197:SSL routines:SSL_shutdown:shutdown while in init']
2020-01-06T15:58:15.058Z mediasoup:ERROR:worker[pid:22] RTC::DtlsTransport::CheckStatus() | OpenSSL error [desc:'SSL status: SSL_ERROR_SSL', error:'error:140E0197:SSL routines:SSL_shutdown:shutdown while in init']
2020-01-06T16:02:57.941Z mediasoup:ERROR:worker[pid:22] (stderr) Assertion failed: current_timestamp_group_.complete_time_ms >= 0 (../deps/libwebrtc/libwebrtc/modules/remote_bitrate_estimator/inter_arrival.cc: BelongsToBurst: 146)
2020-01-06T16:02:57.965Z mediasoup:ERROR:Worker worker process died unexpectedly [pid:22, code:null, signal:SIGABRT]
The browser version we support are:
- chrome/chromium - min 65, recommended 75 (most of our users have at least 74+)
- firefox - min 60, recommended 67 (most of our users have at least 65+)
Currently we do not have core dumps enabled on our production servers. We will enabled them, but this doesn’t happen very often so it might take some time before I can provide more info.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 82 (54 by maintainers)
Commits related to this issue
- Working again in issue #357 — committed to versatica/mediasoup by ibc 4 years ago
- Transport.cpp: enable tccClient on the first video Consumer: - no need for it to be simulcast or SVC. — committed to versatica/mediasoup by ibc 4 years ago
- - libwebrtc Apply patch by @sspanak and @Ivaka to avoid crash. Related issue: #357 - PortManager.cpp: Do not use `UV_UDP_RECVMMSG` in Windows due to a bug in libuv 1.37.0 — committed to versatica/mediasoup by ibc 4 years ago
BTW by looking at the coredump, it’s clear that it’s originated upon receipt of a transport-cc feedback:
this->rtpTransportControllerSend->OnTransportFeedback(*feedback);I saw this on line 327 of the core dump file:
I haven’t dug too deep, nor I have looked at
FeedbackRtpTransport.cpp, so I am not sure ifreceive_timeis the same asreceivedAtMs.Fixed in https://github.com/versatica/mediasoup/pull/985
Here are 4 [ISSUE 357] logs. I hope these are useful.
crash 1
crash 2
crash 3
crash 4
Yes I see:
Regarding 1. Incoming Packet above, I’ve done this: https://github.com/versatica/mediasoup/commit/d8b4a92953d8b8104d1aef48259f3187a19d7579
Regarding 2. Outgoing Packet above, I’ve seen (and fixed) some bugs: https://github.com/versatica/mediasoup/commit/8ab389b4ee224d433b23045e62d1e96d52c24be6
mediasoup 3.5.7 released with the fix. Thanks a lot guys for your help!
Thanks @sspanak, very useful info. Will work on this these days and notify here.