runtime: Kestrel: Segfault when specifying default certificate on Ubuntu 20.04

Is there an existing issue for this?

I have searched the existing issues

Describe the bug

After upgrading an ASP.NET Core project from .NET 6 to .NET 7, the application fails to start with a segmentation fault when specifying a default certificate for Kestrel.

Expected Behavior

Application works with a certificate on .NET 7 same as before on .NET 6.

Steps To Reproduce

Start the application without specifying a certificate
Add the following in appsettings.json:

{
  "Kestrel": {
    "Certificates": {
      "Default": {
        "Path": "/path/to/cert.p12",
        "Password": "password"
      }
    }
  }
}

After the configuration has been reloaded, see Main process exited, code=killed, status=11/SEGV

Alternatively, can just specify the certificate from the start and see that the application crashes before starting.

Exceptions (if any)

I’ve seen two types of segfaults in the hypervisor syslog: after updating appsettings.json

kernel: .NET ThreadPool[3421307]: segfault at 0 ip 00007f32507860be sp 00007f3226bb58c0 error 4 in libcrypto.so.1.1[7f3250674000+19b000]
kernel: Code: 00 00 4c 89 f1 4c 89 fa e8 bf 24 07 00 85 c0 0f 84 bf 00 00 00 8b 54 24 1c 49 8d 7c 24 10 4c 89 fe e8 56 2b f0 ff 85 c0 74 52 <48> 63 75 00 48 8b 7d 08 45 31 c9 49 89 d8 4c 89 f1 4c 89 fa e8 49

and starting the application with default certificates:

kernel: Gnomeshade.WebA[3397350]: segfault at 0 ip 00007f705b6200be sp 00007fff3b39f640 error 4 in libcrypto.so.1.1[7f705b50e000+19b000]
kernel: Code: 00 00 4c 89 f1 4c 89 fa e8 bf 24 07 00 85 c0 0f 84 bf 00 00 00 8b 54 24 1c 49 8d 7c 24 10 4c 89 fe e8 56 2b f0 ff 85 c0 74 52 <48> 63 75 00 48 8b 7d 08 45 31 c9 49 89 d8 4c 89 f1 4c 89 fa e8 49

.NET Version

7.0.102

Anything else?

Running on Ubuntu 20.04.5 LTS LXC container on Proxmox 7.3-4. libbssl version:

~$ sudo apt show libssl1.1
Package: libssl1.1
Version: 1.1.1f-1ubuntu2.16

About this issue

Original URL
State: closed
Created a year ago
Comments: 27 (21 by maintainers)

Most upvoted comments

@Qowy based on the servicing pull request, it looks like it will be merged in for the 7.0.5 servicing release.

vcsjones on Mar 8, 2023

@VMelnalksnis

I just realized I never gave a proper write-up of the issue here, only obtuse debugging notes. Apologies.

The issue you are running in to is due to a new feature in .NET 7 called OCSP stapling. Unfortunately there is an issue with the implementation when handling an X.509 chain that has exactly two certificates in it, an end-entity and a root, which your certificate chain (appears) to have.

This has been fixed for .NET 8, and we are in the process of attempting to get it serviced in to a future .NET 7 release. If or when that happens I can’t say for certain, but I’ll give an update here when a servicing decision has been made.

In the mean time, you have a few options.

Stay on .NET 6 until a .NET version with the fix is available (either .NET 8, or a serviced version of .NET 7)

Disable revocation checking for the server certificate. As far as I can tell, Kestrel’s JSON configuration doesn’t expose the right setting we need, however configuring kestrel in code allows us to do so:

var builder = WebApplication.CreateBuilder(args);

builder.WebHost.ConfigureKestrel(serverOptions => {
    serverOptions.ListenAnyIP(5000, listenOptions => {
        static ValueTask<SslServerAuthenticationOptions> HttpsCallback(
            SslStream stream,
            SslClientHelloInfo clientHelloInfo,
            object? state,
            CancellationToken cancellationToken)
        {
            SslServerAuthenticationOptions options = new()
            {
                // NOTE: hardcoded password for certificate is for illustration purposes only.
                ServerCertificate = new X509Certificate2("/app/leaf.p12", "potato"),
                CertificateRevocationCheckMode = X509RevocationMode.NoCheck,
            };
            return new ValueTask<SslServerAuthenticationOptions>(options);
        }
        listenOptions.UseHttps(HttpsCallback, state: null!);
        listenOptions.Protocols = HttpProtocols.Http1AndHttp2AndHttp3;
    });
});

The important bit there is setting CertificateRevocationCheckMode to NoCheck for the ServerAuthenticationOptions. Essentially what you are doing here is disabling OCSP stapling, along with revocation checking of your server certificate.

Change your CA to issue off of an intermediate X.509 certificate so there are at least three certificates in your chain.

vcsjones on Feb 18, 2023