srt: [BUG] srt_send() hangs forever with high bitrates
Describe the bug
When running SRT client in Live mode, sending packets to srt-live-transmit, at some point the call to srt_send()
hangs forever (i.e. the call never finishes, until at least the socket on the server side is not closed).
My setup is:
- client at host A: custom SRT client side sending data with bitrate of 80+ Mbit (v1.5.2 from master)
- server at host B: srt-live-transmit in a Docker container used as a server side (v1.5.1)
- 50-60 Mbit link speed between client and server hosts
I understand that it’s possibly “too much data” being sent, but I expect for the packets to be dropped and not for everything to be hanging/blocked.
To Reproduce
The minimal class is just sending a lot of data in the loop and checking if the call to srt_send()
blocks. I attached a self contained cpp file.
Steps to reproduce the behavior:
- Run srt-live-transmit in a Docker container in a host B:
srt-live-transmit srt://:6011 srt://:6021
- Run srt-hangs executable in a host A:
srt-hangs 100000000 IP-of-host-B 6011
(srt-hangs bitrate ip port
) - Wait (for me it’s usually up to 1-2 minutes)
srt_send()
is hanging and the following logs are visible:
On the client (A) side:
~# srt-hangs 100000000 12.34.56.78 6011
using SRT version 1.5.2
Opening SRT streamer for 12.34.56.78:6011
SRT streamer opened
SRT watchdog started
send took 475 ms
send took 760 ms
send took 1035 ms
send took 1213 ms
... etc.
send took 1232 ms
SRT send() hangs for 2000 ms
SRT send() hangs for 4000 ms
SRT send() hangs for 6000 ms
... etc.
On the server (B) side:
... a lot of similar to this:
Mar 31 22:19:27 localhost container_name/srt-left[3729]: 22:19:27.485421/SRT:TsbPd!W:SRT.br: @531913161:RCV-DROPPED 3 packet(s). Packet seqno %200848698 delayed for 893.862 ms
Mar 31 22:19:27 localhost container_name/srt-left[3729]: 22:19:27.506651/SRT:TsbPd!W:SRT.br: @531913161:RCV-DROPPED 8 packet(s). Packet seqno %200848815 delayed for 894.256 ms
Mar 31 22:19:27 localhost container_name/srt-left[3729]: 22:19:27.526165/SRT:TsbPd!W:SRT.br: @531913161:RCV-DROPPED 42 packet(s). Packet seqno %200848963 delayed for 894.332 ms
Expected behavior srt_send() should never block forever
Desktop (please provide the following information):
- OS:
- client side A: Linux TinaLinux 4.9.118 armv7l, GCC 6.4.1
- server side B: Linux Ubuntu 20.04.4 4.9.224 aarch64 + Docker image with srt based on ubuntu:18.04
- SRT Version / commit ID:
- client side A: v.1.5.2 from master/39822840,
- server side B: v1.5.1 release
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 27
Commits related to this issue
- Created version for testing hangup #2706 with some fixes — committed to ethouris/srt by deleted user a year ago
- [core] Fix lacking mutex protection of some ACK-related fields (#2723). Repeated fix for #2706. — committed to Haivision/srt by ethouris 9 months ago
Good news! With the changes from the test branch of @ethouris I cannot reproduce the problem anymore!
I tried it “as is”, and then also with logging completely disabled in the test app. I also proceeded to use the obtained
libsrt.a
in my real app and it all looks good! No repro so far!I will only be able to continue investigation beginning Tuesday next week. Thank you for your support, I will report back when I have some news.