sctp: SCTP throughput ~8x slower than TCP when the connection has latency
Your environment.
- Version: sctp v1.8.2
- Other Information: tested between macOS<–>macOS and macOS<–>debian bullseye
What did you do?
Test setup:
- Throughput tested with @enobufs’s sctptest, dependencies updated to use the latest version of pion/sctp
- Latency controlled using macOS “Network Link Conditioner”
- Tested between separate physical devices connected by gigabit ethernet
I ran sctp-test in both TCP and SCTP (sctp-test refers to this as “udp”) modes and compared the throughput with varying levels of latency:
- For TCP:
./sctptest -network tcp -s <addr>[client] and./sctptest -network tcp[server] - For SCTP:
./sctptest -network udp -s <addr>[client] and./sctptest -network udp[server]
What did you expect?
I expected the TCP and SCTP throughput to be in the same ballpark.
What happened?
SCTP overwhelmingly underperformed TCP. Approximate performance over gigabit ethernet:
| Latency | TCP | SCTP |
|---|---|---|
| 2ms | 850 Mbps | 850 Mbps |
| 20ms | 800 Mbps | 200 Mbps |
| 50ms | 600 Mbps | 85 Mbps |
| 100ms | 320 Mbps | 45 Mbps |
| 300ms | 105 Mbps | 17 Mbps |
Background
I ran into this performance issue when trying to diagnose why DataChannel performance was very slow when using a TURN server. My TURN server is about 100ms away, and I was seeing very poor performance (< 10 Mbps) relative to the speed of the internet connection.
https://user-images.githubusercontent.com/87964/160219934-5fadc424-6b2f-49a3-b335-d17da6c25046.mp4
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 2
- Comments: 15 (8 by maintainers)
Commits related to this issue
- Exclude inflight size from bufferedAmount Relates to #218 — committed to pion/sctp by enobufs 2 years ago
- Max recv buf size via SettingEngine Relates to pion/sctp#218 — committed to pion/webrtc by enobufs 2 years ago
- Added a unit test Relates to pion/sctp#218 — committed to pion/webrtc by enobufs 2 years ago
- Max recv buf size via SettingEngine Relates to pion/sctp#218 — committed to pion/webrtc by enobufs 2 years ago
- Added a unit test Relates to pion/sctp#218 — committed to pion/webrtc by enobufs 2 years ago
- Max recv buf size via SettingEngine Relates to pion/sctp#218 — committed to pion/webrtc by enobufs 2 years ago
- Added a unit test Relates to pion/sctp#218 — committed to pion/webrtc by enobufs 2 years ago
- Max recv buf size via SettingEngine Relates to pion/sctp#218 — committed to daonb/webrtc by enobufs 2 years ago
- Added a unit test Relates to pion/sctp#218 — committed to daonb/webrtc by enobufs 2 years ago
With your updated version of go-rudp/sctp-test (passing buffer size into pion/sctp assoc), I see dramatically better throughput.
For the curious:
This is faster than TCP. To repro these tests, I had to make a small fix to cap conn.SetReadBuffer/SetWriteBuffer to ~5mb for mac (the OS UDP limit), while allowing the sctp buffer to be the specified size
Very excited for the use-cases this unlocks for pion/sctp!
@iamcalledrob Thanks for the detailed report. As I mentioned in #62, a modern TCP implementation automatically adjust receive buffer size, while pion/sctp uses a fixed buffer size, which is currently set to 1024KB. The receive buffer size sets the hard limit for the amount of data inflight (or the un-ack’d segments). It appears the throughput you observed is the result of this limit. I think pion/sctp should implement this auto tuning feature. Let me dig into more information about its algorithm.
@enobufs I’m curious if the commit to exclude the in-flight size is ready to land (or could be!)
I’ve been testing datachannels with the larger buffer offered through the SettingEngine, but am still seeing very low throughout when there’s even a tiny amount of packet loss or latency.
Testing with sctptest over LAN and the network link conditioner looked really positive–near 100% link utilisation.
However in the real world I’m still seeing <20% link utilisation over WiFi or coast-to-coast levels of latency.
For example, I have a TURN server which is 80ms away. iperf with a few TCP streams will get 600+Mbps throughput. PION datachannels with an 8mb buffer will get 20Mbps throughput.
I was hoping the larger buffer would be a decent enough fix by itself in the “real world”, but that doesn’t seem to be the case!
@Sean-Der I think it is a brilliant idea to allow changing the send/recv buffer size from the SettingEngine as I think it may take sometime for us to implement auto buffer size tuning.
That’s so exciting! Nice work @iamcalledrob and @enobufs 😃
We should add this to the
SettingEnginetoday and publish the findings!After fixing the above in sctptest, I was still seeing the throughput was still capped at 1MB receive buffer size. (increasing the size beyond 1MB had no effect).
sctptest RTT=300ms
From the trance log, I noticed that even though there’s more room in the congestion window, the inflightQueue was not fully utilized. Then I realized, this was because sctptest has maxBufferedAmount set to 1MB (threshold being 512KB, both hardcoded), which was effectively limiting the send buffer size to 1MB. This is why the throughput did not grow beyond 1MB of receive buffer size.
There was another related issue. The bufferedAmount in the current pion/sctp includes the number of bytes inflight. Looking at the FireFox’s implementation, it does not include the size inflight (the size written to usrsctp).
After fixing the all above, now I see the throughput grows proportionally to the size of buffers as shown below:
Other related metrics:
I will write some unit tests then submit a pull-request.
Once all these fixes land, we should consider whether we should dynamically adjust the receive buffer size.
That indicates there would be another limiting factor. From my quick glance, the sender’s congestion window kept growing during the 10 sec period but it appears too slow - smell of a bug… I may be wrong. Let me look at this later more carefully.