runtime: Slow SSL Authentication on Ubuntu 16.04 / 18.04
We investigate errors during SSL authentication on ubuntu using kestrel.
We initiate many concurrent connections and observer that we hang on AuthenticateAsClientAsync and AuthenticateAsServerAsync.
While trying to make a minimal repo, we decided to do a simple sever-client app that simulate our usage.
Running the same app on windows resulted in: Plain ~20ms SSL ~200ms
while on ubuntu 18.04: Plain ~1s SSL ~40s
I also tried on a VM running 16.04 Plain ~1s SSL ~5s
In all of the test the dotnet version is 2.1.403
using System;
using System.Net;
using System.Diagnostics;
using System.Net.Sockets;
using System.Security.Cryptography.X509Certificates;
using System.Threading;
using System.Threading.Tasks;
namespace ConsoleApp1
{
public static class Program
{
public static void Main(string[] args)
{
#if USE_SSL
System.Console.WriteLine("SSL");
#else
Console.WriteLine("Plain");
#endif
var globalCertificate = new X509Certificate2("path/to/cert.pfx");
var listener = new TcpListener(new IPAddress(new byte[] { 127, 0, 0, 1 }), 56789);
listener.Start();
for (var i = 0; i < 1; i++)
Listen(listener, globalCertificate);
var overall = Stopwatch.StartNew();
int done = 0;
int failed = 0;
var clients = 200;
var tasks = new Task[clients];
for (int i = 0; i < clients; i++)
{
var j = i;
tasks[i] = Task.Run(async () =>
{
var sp = Stopwatch.StartNew();
try
{
using (var tcpClient = new TcpClient())
{
await tcpClient.ConnectAsync(new IPAddress(new byte[] { 127, 0, 0, 1 }), 56789);
#if USE_SSL
using (var wrapped = new SslStream(tcpClient.GetStream(), false, ((sender, certificate, chain, errors) => true)))
{
await wrapped.AuthenticateAsClientAsync("localhost", new X509CertificateCollection(new[] {globalCertificate}),
SslProtocols.Tls12, false);
#else
using (var wrapped = tcpClient.GetStream())
{
#endif
byte[] buffer = { 1 };
await wrapped.WriteAsync(buffer, 0, 1);
var read = await wrapped.ReadAsync(buffer, 0, 1);
if (read != 1)
{
Console.WriteLine("why?");
}
Interlocked.Increment(ref done);
}
}
}
catch (Exception e)
{
Console.WriteLine(Interlocked.Increment(ref failed) + " FAILED " + sp.Elapsed);
Console.WriteLine(e);
}
});
}
Task.WaitAll(tasks);
Console.WriteLine("Done " + done + " failed " + failed + " in " + overall.Elapsed);
}
private static void Listen(TcpListener listener, X509Certificate globalCertificate)
{
Task.Run(async () =>
{
try
{
var tcpClient = await listener.AcceptTcpClientAsync();
Listen(listener, globalCertificate);
#if USE_SSL
using (var wrapped = new SslStream(tcpClient.GetStream(), false, ((sender, certificate, chain, errors) => true)))
{
await wrapped.AuthenticateAsServerAsync(globalCertificate);
#else
using (var wrapped = tcpClient.GetStream())
{
#endif
var buffer = new byte[1];
var read = await wrapped.ReadAsync(buffer, 0, 1);
await wrapped.WriteAsync(buffer, 0, 1);
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
});
}
}
}
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 1
- Comments: 23 (14 by maintainers)
one more benchmark ->
crankform asp.net RPS: 1,018 ->19,928crank compare 6.0.json 7.0.json
This should be fixed in 7.0 running benchmark from https://github.com/dotnet/performance
For TLS resume to work, the server side has to use SslStreamCertificateContext. (and no CipherSuitePolicy)
Re methodology here: I’m always a little skeptical of tests where we spawn a lot (200, here) of tasks at the same time. The problem is that this introduces the added variable of task/thread pool scheduling, which has differences between Windows and Linux.
A better approach, IMO, is to spawn a fixed number of threads and have them loop over many iterations, then measure the overall throughput.
I do think it’s likely we have a problem here, but I’d like to get numbers that better reflect the actual problem without introducing additional variables.
Windows results
Ubuntu 16.04 VM
Real machine Ubuntu 18.04
This is first time I use BenchmarkDotNet so hope I got it correct 😃