ztunnel: Ztunnel does not scale up with number of worker threads in expected way due to h2 locking

Some testinging:

Running locally (i.e. ONE ztunnel connecting to itself)
2 workers
1:  11.51 Gbits/sec
2:  11.92 Gbits/sec
3:  11.33 Gbits/sec
4:  11.30 Gbits/sec
6:  12.15 Gbits/sec
8:  12.77 Gbits/sec
16: 14.95 Gbits/sec

4 workers
1:  9.52 Gbits/sec
2:  15.89 Gbits/sec
3:  16.49 Gbits/sec
4:  17.05 Gbits/sec
6:  16.28 Gbits/sec
8:  16.15 Gbits/sec
16: 15.74 Gbits/sec

8 workers
1:  7.88 Gbits/sec
2:  12.27 Gbits/sec
3:  15.73 Gbits/sec
4:  17.14 Gbits/sec
6:  17.95 Gbits/sec
8:  18.63 Gbits/sec
16: 19.03 Gbits/sec

On Kubernetes, cross node
2 workers
1:  9.27 Gbits/sec
2:  10.11 Gbits/sec
3:  10.60 Gbits/sec
4:  11.72 Gbits/sec
6:  13.55 Gbits/sec
8:  14.34 Gbits/sec
16: 16.09 Gbits/sec

4 workers
1:  8.09 Gbits/sec
2:  9.16 Gbits/sec
3:  9.72 Gbits/sec
4:  11.31 Gbits/sec
6:  11.64 Gbits/sec
8:  13.67 Gbits/sec
16: 15.08 Gbits/sec

8 workers
1:  8.09 Gbits/sec
2:  8.61 Gbits/sec
3:  9.27 Gbits/sec
4:  10.63 Gbits/sec
6:  11.70 Gbits/sec
8:  13.17 Gbits/sec
16: 15.57 Gbits/sec


On kubernetes, raw TCP (not HBONE/TLS)
2 workers
1:  22.18 Gbits/sec
2:  35.72 Gbits/sec
3:  35.36 Gbits/sec
4:  35.28 Gbits/sec
6:  37.48 Gbits/sec
8:  37.37 Gbits/sec
16: 38.62 Gbits/sec

8 workers
1:  23.96 Gbits/sec
2:  34.57 Gbits/sec
3:  43.82 Gbits/sec
4:  51.85 Gbits/sec
6:  50.36 Gbits/sec
8:  47.48 Gbits/sec
16: 43.99 Gbits/sec

Note: I have a 16c machine.

The weird thing is I do not see the same behavior in our benchmarks (with https://github.com/istio/ztunnel/pull/1171)

                        thrpt:  [12.107 Gelem/s 13.392 Gelem/s 14.496 Gelem/s]
                 change:
                        time:   [-3.0115% +5.8051% +18.549%] (p = 0.40 > 0.05)
                        thrpt:  [-15.647% -5.4866% +3.1050%]
                        No change in performance detected.
Found 2 outliers among 10 measurements (20.00%)
  2 (20.00%) high severe
throughput/hbone2       time:   [376.68 ms 384.08 ms 392.21 ms]
                        thrpt:  [21.901 Gelem/s 22.365 Gelem/s 22.804 Gelem/s]
                 change:
                        time:   [-12.720% -9.1954% -5.5575%] (p = 0.00 < 0.05)
                        thrpt:  [+5.8845% +10.127% +14.574%]
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
throughput/hbone8       time:   [501.59 ms 507.84 ms 514.72 ms]
                        thrpt:  [16.689 Gelem/s 16.915 Gelem/s 17.125 Gelem/s]
                 change:
                        time:   [-2.8183% -0.8154% +1.2152%] (p = 0.46 > 0.05)
                        thrpt:  [-1.2006% +0.8221% +2.9000%]
                        No change in performance detected.

Things I have tried with success:

  • H2 Window sizes - no impact
  • Tracing to try to see where we are waiting - inconclusive
  • Disabling pooling, so we have 1:1 connections - no impact

I would expect HBONE to behave like TCP: with 2 workers, we see ~2x throughput with 2 connections as 1. 2x is the theoretical best, of course, so something more like 1.5x-1.8x may be more realistic. We don’t see that. We see virtually zero increase. We do see improvements at very high connection counts, though.

We need to find out what is holding up the processing

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 16 (16 by maintainers)

Most upvoted comments

These tests are “open connection once” then “send a ton of information” on them. The connection establishment is out of the measurement path entirely basically.

I believe this was https://crates.io/crates/netperf (iperf but with parallel connections, I think some iperfs can do parallel as well) and tuning ZTUNNEL_WORKER_THREADS but I forget the exact steps