runtime: cryptography/security test failures on RHEL9/Fedora 36.

Some cryptography/security tests fail on RHEL9 and the next version of Fedora (36). Both these distros have OpenSSL 3.0.

System.Security.Cryptography.Rsa.Tests.ImportExport.PublicOnlyPrivateExport fails with:

<test name="System.Security.Cryptography.Rsa.Tests.ImportExport.PublicOnlyPrivateExport" type="System.Security.Cryptography.Rsa.Tests.ImportExport" method="PublicOnlyPrivateExport" time="0.0074248" result="Fail">
<failure exception-type="Xunit.Sdk.ThrowsException">
<message>
Assert.Throws() Failure\nExpected: typeof(System.Security.Cryptography.CryptographicException)\nActual: typeof(System.OutOfMemoryException): Insufficient memory to continue the execution of the program.\n---- System.OutOfMemoryException : Insufficient memory to continue the execution of the program.
</message>
<stack-trace>
at Interop.Crypto.GetPkcs8PrivateKeySize(IntPtr pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 116 at Interop.Crypto.RentEncodePkcs8PrivateKey(SafeEvpPKeyHandle pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 134 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportPrivateKey[T](ExportPrivateKeyFunc`1 exporter) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 293 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportParameters(Boolean includePrivateParameters) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 430 at System.Security.Cryptography.Rsa.Tests.ImportExport.<>c__DisplayClass11_0.<PublicOnlyPrivateExport>b__0() in /home/tester/runtime/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/ImportExport.cs:line 237 ----- Inner Stack Trace ----- at Interop.Crypto.GetPkcs8PrivateKeySize(IntPtr pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 116 at Interop.Crypto.RentEncodePkcs8PrivateKey(SafeEvpPKeyHandle pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 134 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportPrivateKey[T](ExportPrivateKeyFunc`1 exporter) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 293 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportParameters(Boolean includePrivateParameters) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 430 at System.Security.Cryptography.Rsa.Tests.ImportExport.<>c__DisplayClass11_0.<PublicOnlyPrivateExport>b__0() in /home/tester/runtime/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/ImportExport.cs:line 237
</stack-trace>
</failure>
</test>

Other tests like System.Security.Cryptography.Rsa.Tests.RSAKeyFileTests.NoPrivKeyFromPublicOnly, System.Security.Cryptography.Rsa.Tests.RSAImportExportCspBlobTests.ExportImportPublicOnly, … fail in a similar way: GetPkcs8PrivateKeySize throws OutOfMemoryException instead of CryptographicException.

System.Security.Cryptography.Tests.AsnEncodedDataTests.TestSubjectAlternativeName_Unix fails with:

<test name="System.Security.Cryptography.Tests.AsnEncodedDataTests.TestSubjectAlternativeName_Unix" type="System.Security.Cryptography.Tests.AsnEncodedDataTests" method="TestSubjectAlternativeName_Unix" time="0.0056087" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>
Assert.Equal() Failure\n ↓ (pos 10)\nExpected: othername:<unsupported>, email:sanemail1@example.or···\nActual: othername: UPN::subjectupn1@example.org, email:sane···\n ↑ (pos 10)
</message>
<stack-trace>
at System.Security.Cryptography.Tests.AsnEncodedDataTests.TestSubjectAlternativeName_Unix() in /home/tester/runtime/src/libraries/System.Security.Cryptography/tests/AsnEncodedDataTests.cs:line 136
</stack-trace>
</failure>
</test>

System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail (same for SslStreamAlpnTest_Sync) fails with:

<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail" time="0.4657042" result="Fail">
<failure exception-type="System.AggregateException">
<message>
System.AggregateException : One or more errors occurred. (Authentication failed, see inner exception.) (Authentication failed, see inner exception.)\n---- System.Security.Authentication.AuthenticationException : Authentication failed, see inner exception.\n-------- Interop+OpenSsl+SslException : SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.\n------------ Interop+Crypto+OpenSslCryptographicException : error:0A000460:SSL routines::reason(1120)\n---- System.Security.Authentication.AuthenticationException : Authentication failed, see inner exception.\n-------- Interop+OpenSsl+SslException : SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.\n------------ Interop+Crypto+OpenSslCryptographicException : error:0A0000EB:SSL routines::no application protocol
</message>
<stack-trace>
at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 88 at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55 at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail() in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 168 --- End of stack trace from previous location --- ----- Inner Stack Trace #1 (System.Security.Authentication.AuthenticationException) ----- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 418 at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120 ----- Inner Stack Trace ----- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 384 at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 161 ----- Inner Stack Trace ----- ----- Inner Stack Trace #2 (System.Security.Authentication.AuthenticationException) ----- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 418 at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120 ----- Inner Stack Trace ----- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 384 at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 161 ----- Inner Stack Trace -----
</stack-trace>
</failure>
</test>

System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success (same for SslStreamAlpnTest_Sync) fails for a number of combinations:

<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success(clientProtocols: [http/1.1, h2], serverProtocols: [h2], expected: )" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_Success" time="0.4008237" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>Assert.Equal() Failure\nExpected: \nActual: h2</message>
<stack-trace>
at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_Success(List`1 clientProtocols, List`1 serverProtocols, SslApplicationProtocol expected) in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 127 --- End of stack trace from previous location ---
</stack-trace>
</failure>
</test>
<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success(clientProtocols: [http/1.1], serverProtocols: [http/1.1, h2], expected: )" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_Success" time="0.3577751" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>
Assert.Equal() Failure\nExpected: \nActual: http/1.1
</message>
<stack-trace>
at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_Success(List`1 clientProtocols, List`1 serverProtocols, SslApplicationProtocol expected) in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 127 --- End of stack trace from previous location ---
</stack-trace>
</failure>
</test>
<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success(clientProtocols: [http/1.1, h2], serverProtocols: [http/1.1, h2], expected: )" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_Success" time="0.4638295" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>
Assert.Equal() Failure\nExpected: \nActual: http/1.1
</message>
<stack-trace>
at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_Success(List`1 clientProtocols, List`1 serverProtocols, SslApplicationProtocol expected) in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 127 --- End of stack trace from previous location ---
</stack-trace>
</failure>
</test>

cc @bartonjs @omajid

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 21 (21 by maintainers)

Most upvoted comments

Good news. It’s a test bug. The tests believe that ALPN isn’t available, because they did a bad version check instead of a capabilities check. So what they’re reporting is “ALPN wasn’t supposed to work, but it did!”

I fixed the version check locally, and:

$ LD_PRELOAD="/usr/local/lib/libcrypto.so.3 /usr/local/lib/libssl.so.3" /home/jbarton/git/bartonjs/runtime/artifacts/bin/testhost/net6.0-Linux-Debug-x64/dotnet exec --runtimeconfig System.Net.Security.Tests.runtimeconfig.json --depsfile System.Net.Security.Tests.deps.json xunit.console.dll System.Net.Security.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  Discovering: System.Net.Security.Tests (method display = ClassAndMethod, method display options = None)
3.0.0
  Discovered:  System.Net.Security.Tests (found 287 of 472 test cases)
  Starting:    System.Net.Security.Tests (parallel test collections = on, max threads = 2)
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_SelfSignedClientEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_ServerEKUClientAuth_Fails [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_ClientEKUServerAuth_Fails [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUServerAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
  Finished:    System.Net.Security.Tests
=== TEST EXECUTION SUMMARY ===
   System.Net.Security.Tests  Total: 4734, Errors: 0, Failed: 0, Skipped: 5, Time: 25.289s

bartonjs on Jan 19, 2022

After upgrading, same story. Which means that I wasn’t actually getting 3.0 loaded.

I see the failures now.

bartonjs on Jan 19, 2022

Well, I meant more “If the error is not ERR_R_MALLOC_FAILURE, treat it as ‘no private key’.”

Ah. Okay, I think I can get that fixed up in the next day or two.

vcsjones on Jan 12, 2022

Oh, it’s PKCS8_pkey_set0 that failed?

It seems I misremembered the part of the if failed for something I checked only 2 hours ago 😄. You’re right. That also lines up with the error queue.

vcsjones on Jan 11, 2022

any chance we are trying to allocate negative number?

It’s not us allocating, it’s OpenSSL setting the error here:

https://github.com/openssl/openssl/blob/e5fb4b1469f317aa92768cdf804dfa29b72cb8f3/providers/implementations/encode_decode/encode_key2any.c#L94

    if ((p8info = PKCS8_PRIV_KEY_INFO_new()) == NULL
        || (derlen = k2d(key, &der)) <= 0
        || !PKCS8_pkey_set0(p8info, OBJ_nid2obj(key_nid), 0,
                            params_type, params, der, derlen)) {
        ERR_raise(ERR_LIB_PROV, ERR_R_MALLOC_FAILURE); // <-- Error is set here

I’m guessing the ERR_R_MALLOC_FAILURE makes sense if p8info is null, but it’s the PKCS8_pkey_set0 that fails and it lumps that in with the allocation failure.

vcsjones on Jan 11, 2022