amazon-kinesis-video-streams-webrtc-sdk-c: [BUG] Crash: corrupted size vs. prev_size

Describe the bug I pulled master, and now the kvsWebrtcClientMaster example crashes once a viewer client connects.
SDK version number commit 4461cf2ba3c16aa3a75bb967c4ce658ce4aa17c8 Date: Wed Jul 22 12:22:56 2020 -0700
To Reproduce Steps to reproduce the behavior
Complete a fresh build of master.
Launch the kvsWebrtcClientMaster example.
Connect a web viewer from the KVS webrtc test page.
You’ll then get some kind of crash in kvsWebrtcClientMaster (see console output below)
Expected behavior I would expect it to start streaming and not crash.
Desktop (please complete the following information):
OS: Ubuntu Xenial 16.04
Additional context console output
2020-07-22 20:45:20 DEBUG   iceAgentLogNewCandidate(): New remote ice candidate discovered. Id: ttus2h7sP. Ip: 54.190.53.169:56874. Type: host. Protocol: UDP.
2020-07-22 20:45:20 INFO    onSctpInboundPacket(): Unhandled PPID on incoming SCTP message 0
2020-07-22 20:45:20 INFO    onDataChannel(): New DataChannel has been opened kvsDataChannel 

2020-07-22 20:45:20 DEBUG   turnConnectionHandleStun(): TURN Allocation succeeded. Life time: 600 seconds. Allocation expiration epoch 159545132068996
2020-07-22 20:45:20 DEBUG   dtlsSessionChangeState(): DTLS init completed. Time taken 300 ms
2020-07-22 20:45:21 DEBUG   onRtcpPacket(): RTCP_PACKET_TYPE_RECEIVER_REPORT 1119271623 692752021 loss: 0 0 seq: 24 jit: 186 lsr: 0 dlsr: 0
2020-07-22 20:45:21 DEBUG   onRtcpPacket(): RTCP_PACKET_TYPE_RECEIVER_REPORT 1119271623 692752021 loss: 0 0 seq: 60 jit: 253 lsr: 0 dlsr: 0
2020-07-22 20:45:22 DEBUG   onRtcpPacket(): RTCP_PACKET_TYPE_SENDER_REPORT 1119271623 16339943326756260861 rtpTs: 3970597922 0 pkts 0 bytes
2020-07-22 20:45:22 DEBUG   onRtcpPacket(): unhandled packet type RTCP_PACKET_TYPE_SOURCE_DESCRIPTION
2020-07-22 20:45:22 DEBUG   onRtcpPacket(): RTCP_PACKET_TYPE_RECEIVER_REPORT 1119271623 692752021 loss: 0 0 seq: 131 jit: 124 lsr: 0 dlsr: 0
2020-07-22 20:45:23 DEBUG   onRtcpPacket(): RTCP_PACKET_TYPE_RECEIVER_REPORT 4195875351 1267296657 loss: 0 0 seq: 120 jit: 16 lsr: 0 dlsr: 0
2020-07-22 20:45:23 DEBUG   onRtcpPacket(): RTCP_PACKET_TYPE_RECEIVER_REPORT 1119271623 692752021 loss: 0 0 seq: 175 jit: 91 lsr: 0 dlsr: 0
2020-07-22 20:45:23 DEBUG   rtcpReportsCallback(): rtcpReportsCallback 15954507234488067 ssrc: 692752021 rtxssrc: 642266226
2020-07-22 20:45:23 DEBUG   rtcpReportsCallback(): sender report 692752021 16339943332822449906 15952956598997086 : 179 packets 85037 bytes
2020-07-22 20:45:23 DEBUG   rtcpReportsCallback(): next sender report 692752021 in 102 msec
2020-07-22 20:45:23 DEBUG   rtcpReportsCallback(): rtcpReportsCallback 15954507234489943 ssrc: 1267296657 rtxssrc: 2045070856
2020-07-22 20:45:23 DEBUG   rtcpReportsCallback(): sender report 1267296657 16339943332823255642 15953557250600909 : 140 packets 22400 bytes
2020-07-22 20:45:23 DEBUG   rtcpReportsCallback(): next sender report 1267296657 in 260 msec
*** Error in `./kvsWebrtcClientMaster': corrupted size vs. prev_size: 0x00007f8c9801e430 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f8cae8b87e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x7e9dc)[0x7f8cae8bf9dc]
/lib/x86_64-linux-gnu/libc.so.6(+0x81cde)[0x7f8cae8c2cde]
/lib/x86_64-linux-gnu/libc.so.6(__libc_calloc+0xba)[0x7f8cae8c5dca]
/home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcClient.so(defaultMemCalloc+0x23)[0x7f8caee6faff]
/home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcClient.so(createStunPacket+0x5b)[0x7f8caee6e3a9]
/home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcClient.so(iceAgentSendSrflxCandidateRequest+0x73)[0x7f8caee4bdbd]
/home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcClient.so(iceAgentGatherCandidateTimerCallback+0x2c6)[0x7f8caee4c715]
/home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcClient.so(timerQueueExecutor+0x1cf)[0x7f8caee756be]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f8cae62b6ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f8cae94841d]
======= Memory map: ========
00400000-0040b000 r-xp 00000000 fd:01 55841981                           /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/kvsWebrtcClientMaster
0060a000-0060b000 r--p 0000a000 fd:01 55841981                           /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/kvsWebrtcClientMaster
0060b000-0060c000 rw-p 0000b000 fd:01 55841981                           /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/kvsWebrtcClientMaster
0164d000-016b1000 rw-p 00000000 00:00 0                                  [heap]
7f8c7c000000-7f8c7c021000 rw-p 00000000 00:00 0 
7f8c7c021000-7f8c80000000 ---p 00000000 00:00 0 
7f8c84000000-7f8c84021000 rw-p 00000000 00:00 0 
7f8c84021000-7f8c88000000 ---p 00000000 00:00 0 
7f8c88000000-7f8c88021000 rw-p 00000000 00:00 0 
7f8c88021000-7f8c8c000000 ---p 00000000 00:00 0 
7f8c8c000000-7f8c8c021000 rw-p 00000000 00:00 0 
7f8c8c021000-7f8c90000000 ---p 00000000 00:00 0 
7f8c917fb000-7f8c917fc000 ---p 00000000 00:00 0 
7f8c917fc000-7f8c91ffc000 rw-p 00000000 00:00 0 
7f8c91ffc000-7f8c91ffd000 ---p 00000000 00:00 0 
7f8c91ffd000-7f8c927fd000 rw-p 00000000 00:00 0 
7f8c927fd000-7f8c927fe000 ---p 00000000 00:00 0 
7f8c927fe000-7f8c92ffe000 rw-p 00000000 00:00 0 
7f8c92ffe000-7f8c92fff000 ---p 00000000 00:00 0 
7f8c92fff000-7f8c937ff000 rw-p 00000000 00:00 0 
7f8c937ff000-7f8c93800000 ---p 00000000 00:00 0 
7f8c93800000-7f8c94000000 rw-p 00000000 00:00 0 
7f8c94000000-7f8c9403f000 rw-p 00000000 00:00 0 
7f8c9403f000-7f8c98000000 ---p 00000000 00:00 0 
7f8c98000000-7f8c980ef000 rw-p 00000000 00:00 0 
7f8c980ef000-7f8c9c000000 ---p 00000000 00:00 0 
7f8c9c000000-7f8c9c021000 rw-p 00000000 00:00 0 
7f8c9c021000-7f8ca0000000 ---p 00000000 00:00 0 
7f8ca37ff000-7f8ca3800000 ---p 00000000 00:00 0 
7f8ca3800000-7f8ca4000000 rw-p 00000000 00:00 0 
7f8ca4000000-7f8ca4021000 rw-p 00000000 00:00 0 
7f8ca4021000-7f8ca8000000 ---p 00000000 00:00 0 
7f8ca8326000-7f8ca833c000 r-xp 00000000 fd:01 54006232                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7f8ca833c000-7f8ca853b000 ---p 00016000 fd:01 54006232                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7f8ca853b000-7f8ca853c000 rw-p 00015000 fd:01 54006232                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7f8ca853c000-7f8ca853d000 ---p 00000000 00:00 0 
7f8ca853d000-7f8ca8d3d000 rw-p 00000000 00:00 0 
7f8ca8d3d000-7f8ca8d3e000 ---p 00000000 00:00 0 
7f8ca8d3e000-7f8ca953e000 rw-p 00000000 00:00 0 
7f8ca953e000-7f8ca953f000 ---p 00000000 00:00 0 
7f8ca953f000-7f8ca9d3f000 rw-p 00000000 00:00 0 
7f8ca9d3f000-7f8ca9d40000 ---p 00000000 00:00 0 
7f8ca9d40000-7f8caa540000 rw-p 00000000 00:00 0 
7f8caa540000-7f8caa557000 r-xp 00000000 fd:01 54013448                   /lib/x86_64-linux-gnu/libresolv-2.23.so
7f8caa557000-7f8caa757000 ---p 00017000 fd:01 54013448                   /lib/x86_64-linux-gnu/libresolv-2.23.so
7f8caa757000-7f8caa758000 r--p 00017000 fd:01 54013448                   /lib/x86_64-linux-gnu/libresolv-2.23.so
7f8caa758000-7f8caa759000 rw-p 00018000 fd:01 54013448                   /lib/x86_64-linux-gnu/libresolv-2.23.so
7f8caa759000-7f8caa75b000 rw-p 00000000 00:00 0 
7f8caa75b000-7f8caa760000 r-xp 00000000 fd:01 54013444                   /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f8caa760000-7f8caa960000 ---p 00005000 fd:01 54013444                   /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f8caa960000-7f8caa961000 r--p 00005000 fd:01 54013444                   /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f8caa961000-7f8caa962000 rw-p 00006000 fd:01 54013444                   /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f8caa962000-7f8caa964000 r-xp 00000000 fd:01 54006299                   /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f8caa964000-7f8caab63000 ---p 00002000 fd:01 54006299                   /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f8caab63000-7f8caab64000 r--p 00001000 fd:01 54006299                   /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f8caab64000-7f8caab65000 rw-p 00002000 fd:01 54006299                   /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f8caab65000-7f8caab70000 r-xp 00000000 fd:01 54013452                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f8caab70000-7f8caad6f000 ---p 0000b000 fd:01 54013452                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f8caad6f000-7f8caad70000 r--p 0000a000 fd:01 54013452                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f8caad70000-7f8caad71000 rw-p 0000b000 fd:01 54013452                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f8caad71000-7f8caad77000 rw-p 00000000 00:00 0 
7f8caad77000-7f8caad78000 ---p 00000000 00:00 0 
7f8caad78000-7f8cab578000 rw-p 00000000 00:00 0 
7f8cab578000-7f8cab579000 ---p 00000000 00:00 0 
7f8cab579000-7f8cabd79000 rw-p 00000000 00:00 0 
7f8cabd79000-7f8cabd7a000 ---p 00000000 00:00 0 
7f8cabd7a000-7f8cac57a000 rw-p 00000000 00:00 0 
7f8cac57a000-7f8cac57b000 ---p 00000000 00:00 0 
7f8cac57b000-7f8cacd7b000 rw-p 00000000 00:00 0 
7f8cacd7b000-7f8cacd7c000 ---p 00000000 00:00 0 
7f8cacd7c000-7f8cad57c000 rw-p 00000000 00:00 0 
7f8cad57c000-7f8cad57f000 r-xp 00000000 fd:01 54013083                   /lib/x86_64-linux-gnu/libdl-2.23.so
7f8cad57f000-7f8cad77e000 ---p 00003000 fd:01 54013083                   /lib/x86_64-linux-gnu/libdl-2.23.so
7f8cad77e000-7f8cad77f000 r--p 00002000 fd:01 54013083                   /lib/x86_64-linux-gnu/libdl-2.23.so
7f8cad77f000-7f8cad780000 rw-p 00003000 fd:01 54013083                   /lib/x86_64-linux-gnu/libdl-2.23.so
7f8cad780000-7f8cad7b8000 r-xp 00000000 fd:01 55841910                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libwebsockets.so.15
7f8cad7b8000-7f8cad9b8000 ---p 00038000 fd:01 55841910                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libwebsockets.so.15
7f8cad9b8000-7f8cad9b9000 r--p 00038000 fd:01 55841910                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libwebsockets.so.15
7f8cad9b9000-7f8cad9ba000 rw-p 00039000 fd:01 55841910                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libwebsockets.so.15
7f8cad9ba000-7f8cada82000 r-xp 00000000 fd:01 55840550                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libusrsctp.so.1.0.0
7f8cada82000-7f8cadc81000 ---p 000c8000 fd:01 55840550                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libusrsctp.so.1.0.0
7f8cadc81000-7f8cadc82000 r--p 000c7000 fd:01 55840550                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libusrsctp.so.1.0.0
7f8cadc82000-7f8cadc84000 rw-p 000c8000 fd:01 55840550                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libusrsctp.so.1.0.0
7f8cadc84000-7f8cadc85000 rw-p 00000000 00:00 0 
7f8cadc85000-7f8cadc9c000 r-xp 00000000 fd:01 55843399                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libsrtp2.so
7f8cadc9c000-7f8cade9b000 ---p 00017000 fd:01 55843399                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libsrtp2.so
7f8cade9b000-7f8cade9c000 r--p 00016000 fd:01 55843399                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libsrtp2.so
7f8cade9c000-7f8cade9d000 rw-p 00017000 fd:01 55843399                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libsrtp2.so
7f8cade9d000-7f8cae15e000 r-xp 00000000 fd:01 55868534                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libcrypto.so.1.1
7f8cae15e000-7f8cae35e000 ---p 002c1000 fd:01 55868534                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libcrypto.so.1.1
7f8cae35e000-7f8cae389000 r--p 002c1000 fd:01 55868534                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libcrypto.so.1.1
7f8cae389000-7f8cae38b000 rw-p 002ec000 fd:01 55868534                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libcrypto.so.1.1
7f8cae38b000-7f8cae390000 rw-p 00000000 00:00 0 
7f8cae390000-7f8cae417000 r-xp 00000000 fd:01 55871368                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libssl.so.1.1
7f8cae417000-7f8cae617000 ---p 00087000 fd:01 55871368                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libssl.so.1.1
7f8cae617000-7f8cae61f000 r--p 00087000 fd:01 55871368                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libssl.so.1.1
7f8cae61f000-7f8cae624000 rw-p 0008f000 fd:01 55871368                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libssl.so.1.1
7f8cae624000-7f8cae63c000 r-xp 00000000 fd:01 54013085                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7f8cae63c000-7f8cae83b000 ---p 00018000 fd:01 54013085                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7f8cae83b000-7f8cae83c000 r--p 00017000 fd:01 54013085                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7f8cae83c000-7f8cae83d000 rw-p 00018000 fd:01 54013085                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7f8cae83d000-7f8cae841000 rw-p 00000000 00:00 0 
7f8cae841000-7f8caea01000 r-xp 00000000 fd:01 54013455                   /lib/x86_64-linux-gnu/libc-2.23.so
7f8caea01000-7f8caec01000 ---p 001c0000 fd:01 54013455                   /lib/x86_64-linux-gnu/libc-2.23.so
7f8caec01000-7f8caec05000 r--p 001c0000 fd:01 54013455                   /lib/x86_64-linux-gnu/libc-2.23.so
7f8caec05000-7f8caec07000 rw-p 001c4000 fd:01 54013455                   /lib/x86_64-linux-gnu/libc-2.23.so
7f8caec07000-7f8caec0b000 rw-p 00000000 00:00 0 
7f8caec0b000-7f8caec32000 r-xp 00000000 fd:01 55841802                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcSignalingClient.so
7f8caec32000-7f8caee31000 ---p 00027000 fd:01 55841802                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcSignalingClient.so
7f8caee31000-7f8caee32000 r--p 00026000 fd:01 55841802                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcSignalingClient.so
7f8caee32000-7f8caee33000 rw-p 00027000 fd:01 55841802                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcSignalingClient.so
7f8caee33000-7f8caee82000 r-xp 00000000 fd:01 55841973                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcClient.so
7f8caee82000-7f8caf081000 ---p 0004f000 fd:01 55841973                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcClient.so
7f8caf081000-7f8caf082000 r--p 0004e000 fd:01 55841973                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcClient.so
7f8caf082000-7f8caf084000 rw-p 0004f000 fd:01 55841973                   /home/matt/amazon-kinesis-video-streams-webrtc-sdk-c/build/libkvsWebrtcClient.so
7f8caf084000-7f8caf0aa000 r-xp 00000000 fd:01 54013084                   /lib/x86_64-linux-gnu/ld-2.23.so
7f8caf10b000-7f8caf279000 rw-p 00000000 00:00 0 
7f8caf286000-7f8caf2a9000 rw-p 00000000 00:00 0 
7f8caf2a9000-7f8caf2aa000 r--p 00025000 fd:01 54013084                   /lib/x86_64-linux-gnu/ld-2.23.so
7f8caf2aa000-7f8caf2ab000 rw-p 00026000 fd:01 54013084                   /lib/x86_64-linux-gnu/ld-2.23.so
7f8caf2ab000-7f8caf2ac000 rw-p 00000000 00:00 0 
7ffffe592000-7ffffe5b4000 rw-p 00000000 00:00 0                          [stack]
7ffffe5e2000-7ffffe5e5000 r--p 00000000 00:00 0                          [vvar]
7ffffe5e5000-7ffffe5e7000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)
About this issue

Original URL
State: closed
Created 4 years ago
Comments: 21 (12 by maintainers)
Most upvoted comments

@mdonahoe you could go ahead and close the ticket - it will still be searchable for others to look.
To answer your question. In this particular case you did the heavy lifting by reporting the issue and almost pinpointing the root cause by bisecting the commits. Big thanks to @disa6302 and @lherman-cs for then scrutinizing the code changes. We have a few tools in our arsenal, including ability to replace the default allocators with the instrumented ones that can have guard-banding around the allocation and will immediately fault the moment the memory actually gets corrupted. This one was a simpler case with code inspection though.
MushMal on Jul 23, 2020