runtime: Curl exception: Problem with the SSL CA cert (path? access rights?) on centos, fedora, redhat.

The issue seems to be concurrency-related, and affects arbitrary .NET Core applications. With this program, I’m able to reproduce the problem about 25% of the time, using a 4-core CentOS 7 VM.

Decreasing the number of threads seems to make the problem happen less frequently, with the problem not reproducible at all with only one thread. Increasing the number of threads did not significantly increase probability.

using System;
using System.Linq;
using System.Net;
using System.Threading;

namespace ConsoleApp2
{
    class Program
    {
        const string URI = "https://example.com";
        static int Main(string[] args)
        {
            int Result = 0;
            var Workers = Enumerable.Range(0, 4)
                .Select(I => new Thread(() =>
                {
                    try
                    {
                        var Req = WebRequest.CreateHttp(URI);
                        Req.Method = "HEAD";
                        using ( Req.GetResponseAsync().Result ) ;
                    }
                    catch ( Exception ex )
                    {
                        Console.WriteLine($"{I}: {ex}");
                        Result = 1;
                    }
                })
                { IsBackground = true })
                .ToArray();
            foreach ( var T in Workers )
                T.Start();
            foreach ( var T in Workers )
                T.Join();
            return Result;
        }
    }
}

System/version information:

# uname -a
Linux frqaclapp01.expoexchange.com 3.10.0-514.26.2.el7.x86_64 dotnet/corefx#1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

# dotnet --version
1.0.4

# curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.21 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

# rpm -qa | perl -ne 'm/curl|nss\b/&&print'
curl-7.29.0-35.el7.centos.x86_64
nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64
nss-softokn-3.16.2.3-14.4.el7.x86_64
nss-sysinit-3.28.4-1.2.el7_3.x86_64
nss-tools-3.28.4-1.2.el7_3.x86_64
libcurl-7.29.0-35.el7.centos.x86_64
python-pycurl-7.19.0-19.el7.x86_64
nss-softokn-freebl-3.16.2.3-14.4.el7.i686
nss-util-3.28.4-1.0.el7_3.x86_64
nss-3.28.4-1.2.el7_3.x86_64

[EDIT] Add C# syntax highlight by @karelz

outline the steps taken to properly compile it and generate it ourselves.

That sounds very reasonable - I am waiting on @tmds or @Priya91 to provide more details here …

@jaredrsowers

bundling curl with coreclr would be a more robust “workaround”

While you are correct it also comes with huge burden and cost - see https://github.com/dotnet/corefx/issues/16201#issuecomment-288769690. We considered this approach (also from other reasons), but rejected it - see dotnet/corefx#17647.

@tmds @Priya91 can we document clearly how to workaround this problem on older CentOS versions? If we see more customers hitting the problem, we should probably mention it in release notes or supported OS doc

I’m having the issue using 1.1 on CentOS 7, and I’m having trouble trying the workaround as suggested by @Priya91 and @tmds. I can set the LD_LIBRARY_PATH env var with a path that contains a libcurl built against openssl, and have confirmed in the code it’s getting it with an Environment.GetEnvironmentVariable(“LD_LIBRARY_PATH”) call dumping to the console. And yet, when I enable event tracing I can see the debug message from curlhandler https://github.com/dotnet/corefx/blob/108260a51b52d40c848c99dd903e2e1de4d9eb62/src/System.Net.Http/src/System/Net/Http/Unix/CurlHandler.cs#L172 confirming it’s getting the NSS based one

Does the path variable override only work with the 2.0 runtime?

failed again in ci, detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_centos7.1_release/45/testReport/System.Net.Http.Functional.Tests/HttpClientHandler_SslProtocols_Test/GetAsync_SupportedSSLVersion_Succeeds_sslProtocols__Tls__url____https___www_ssllabs_com_10301____/

Made the change to provide empty cert dir with SSL_DIR env variable. Closing as fixed, please re-open if failures reappear.

@tmds We may not need to fix this, if dotnet/corefx#17647 happens. Pushing this out for now, to focus on other 2.0 work items. Will revisit in some time again. Thanks for being on top of this 😃

@Priya91 don’t know if you saw I provided some additional packages. @Priya91 @karelz @stephentoub The packages built by Red Hat for RHEL use OpenSSL.