runtime: cryptography/security test failures on RHEL9/Fedora 36.

Some cryptography/security tests fail on RHEL9 and the next version of Fedora (36). Both these distros have OpenSSL 3.0.

System.Security.Cryptography.Rsa.Tests.ImportExport.PublicOnlyPrivateExport fails with:

<test name="System.Security.Cryptography.Rsa.Tests.ImportExport.PublicOnlyPrivateExport" type="System.Security.Cryptography.Rsa.Tests.ImportExport" method="PublicOnlyPrivateExport" time="0.0074248" result="Fail">
<failure exception-type="Xunit.Sdk.ThrowsException">
<message>
Assert.Throws() Failure\nExpected: typeof(System.Security.Cryptography.CryptographicException)\nActual: typeof(System.OutOfMemoryException): Insufficient memory to continue the execution of the program.\n---- System.OutOfMemoryException : Insufficient memory to continue the execution of the program.
</message>
<stack-trace>
at Interop.Crypto.GetPkcs8PrivateKeySize(IntPtr pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 116 at Interop.Crypto.RentEncodePkcs8PrivateKey(SafeEvpPKeyHandle pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 134 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportPrivateKey[T](ExportPrivateKeyFunc`1 exporter) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 293 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportParameters(Boolean includePrivateParameters) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 430 at System.Security.Cryptography.Rsa.Tests.ImportExport.<>c__DisplayClass11_0.<PublicOnlyPrivateExport>b__0() in /home/tester/runtime/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/ImportExport.cs:line 237 ----- Inner Stack Trace ----- at Interop.Crypto.GetPkcs8PrivateKeySize(IntPtr pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 116 at Interop.Crypto.RentEncodePkcs8PrivateKey(SafeEvpPKeyHandle pkey) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.cs:line 134 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportPrivateKey[T](ExportPrivateKeyFunc`1 exporter) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 293 at System.Security.Cryptography.RSAImplementation.RSAOpenSsl.ExportParameters(Boolean includePrivateParameters) in /home/tester/runtime/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs:line 430 at System.Security.Cryptography.Rsa.Tests.ImportExport.<>c__DisplayClass11_0.<PublicOnlyPrivateExport>b__0() in /home/tester/runtime/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/ImportExport.cs:line 237
</stack-trace>
</failure>
</test>

Other tests like System.Security.Cryptography.Rsa.Tests.RSAKeyFileTests.NoPrivKeyFromPublicOnly, System.Security.Cryptography.Rsa.Tests.RSAImportExportCspBlobTests.ExportImportPublicOnly, … fail in a similar way: GetPkcs8PrivateKeySize throws OutOfMemoryException instead of CryptographicException.

System.Security.Cryptography.Tests.AsnEncodedDataTests.TestSubjectAlternativeName_Unix fails with:

<test name="System.Security.Cryptography.Tests.AsnEncodedDataTests.TestSubjectAlternativeName_Unix" type="System.Security.Cryptography.Tests.AsnEncodedDataTests" method="TestSubjectAlternativeName_Unix" time="0.0056087" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>
Assert.Equal() Failure\n ↓ (pos 10)\nExpected: othername:<unsupported>, email:sanemail1@example.or···\nActual: othername: UPN::subjectupn1@example.org, email:sane···\n ↑ (pos 10)
</message>
<stack-trace>
at System.Security.Cryptography.Tests.AsnEncodedDataTests.TestSubjectAlternativeName_Unix() in /home/tester/runtime/src/libraries/System.Security.Cryptography/tests/AsnEncodedDataTests.cs:line 136
</stack-trace>
</failure>
</test>

System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail (same for SslStreamAlpnTest_Sync) fails with:

<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail" time="0.4657042" result="Fail">
<failure exception-type="System.AggregateException">
<message>
System.AggregateException : One or more errors occurred. (Authentication failed, see inner exception.) (Authentication failed, see inner exception.)\n---- System.Security.Authentication.AuthenticationException : Authentication failed, see inner exception.\n-------- Interop+OpenSsl+SslException : SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.\n------------ Interop+Crypto+OpenSslCryptographicException : error:0A000460:SSL routines::reason(1120)\n---- System.Security.Authentication.AuthenticationException : Authentication failed, see inner exception.\n-------- Interop+OpenSsl+SslException : SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.\n------------ Interop+Crypto+OpenSslCryptographicException : error:0A0000EB:SSL routines::no application protocol
</message>
<stack-trace>
at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 88 at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55 at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail() in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 168 --- End of stack trace from previous location --- ----- Inner Stack Trace #1 (System.Security.Authentication.AuthenticationException) ----- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 418 at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120 ----- Inner Stack Trace ----- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 384 at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 161 ----- Inner Stack Trace ----- ----- Inner Stack Trace #2 (System.Security.Authentication.AuthenticationException) ----- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 418 at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /home/tester/runtime/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120 ----- Inner Stack Trace ----- at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount) in /home/tester/runtime/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 384 at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /home/tester/runtime/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 161 ----- Inner Stack Trace -----
</stack-trace>
</failure>
</test>

System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success (same for SslStreamAlpnTest_Sync) fails for a number of combinations:

<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success(clientProtocols: [http/1.1, h2], serverProtocols: [h2], expected: )" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_Success" time="0.4008237" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>Assert.Equal() Failure\nExpected: \nActual: h2</message>
<stack-trace>
at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_Success(List`1 clientProtocols, List`1 serverProtocols, SslApplicationProtocol expected) in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 127 --- End of stack trace from previous location ---
</stack-trace>
</failure>
</test>
<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success(clientProtocols: [http/1.1], serverProtocols: [http/1.1, h2], expected: )" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_Success" time="0.3577751" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>
Assert.Equal() Failure\nExpected: \nActual: http/1.1
</message>
<stack-trace>
at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_Success(List`1 clientProtocols, List`1 serverProtocols, SslApplicationProtocol expected) in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 127 --- End of stack trace from previous location ---
</stack-trace>
</failure>
</test>
<test name="System.Net.Security.Tests.SslStreamAlpnTest_Async.SslStream_StreamToStream_Alpn_Success(clientProtocols: [http/1.1, h2], serverProtocols: [http/1.1, h2], expected: )" type="System.Net.Security.Tests.SslStreamAlpnTest_Async" method="SslStream_StreamToStream_Alpn_Success" time="0.4638295" result="Fail">
<failure exception-type="Xunit.Sdk.EqualException">
<message>
Assert.Equal() Failure\nExpected: \nActual: http/1.1
</message>
<stack-trace>
at System.Net.Security.Tests.SslStreamAlpnTestBase.SslStream_StreamToStream_Alpn_Success(List`1 clientProtocols, List`1 serverProtocols, SslApplicationProtocol expected) in /home/tester/runtime/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamAlpnTests.cs:line 127 --- End of stack trace from previous location ---
</stack-trace>
</failure>
</test>

cc @bartonjs @omajid

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 21 (21 by maintainers)

Most upvoted comments

Good news. It’s a test bug. The tests believe that ALPN isn’t available, because they did a bad version check instead of a capabilities check. So what they’re reporting is “ALPN wasn’t supposed to work, but it did!”

I fixed the version check locally, and:

$ LD_PRELOAD="/usr/local/lib/libcrypto.so.3 /usr/local/lib/libssl.so.3" /home/jbarton/git/bartonjs/runtime/artifacts/bin/testhost/net6.0-Linux-Debug-x64/dotnet exec --runtimeconfig System.Net.Security.Tests.runtimeconfig.json --depsfile System.Net.Security.Tests.deps.json xunit.console.dll System.Net.Security.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  Discovering: System.Net.Security.Tests (method display = ClassAndMethod, method display options = None)
3.0.0
  Discovered:  System.Net.Security.Tests (found 287 of 472 test cases)
  Starting:    System.Net.Security.Tests (parallel test collections = on, max threads = 2)
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_SelfSignedClientEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_ServerEKUClientAuth_Fails [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_ClientEKUServerAuth_Fails [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUServerAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
  Finished:    System.Net.Security.Tests
=== TEST EXECUTION SUMMARY ===
   System.Net.Security.Tests  Total: 4734, Errors: 0, Failed: 0, Skipped: 5, Time: 25.289s

After upgrading, same story. Which means that I wasn’t actually getting 3.0 loaded.

I see the failures now.

Well, I meant more “If the error is not ERR_R_MALLOC_FAILURE, treat it as ‘no private key’.”

Ah. Okay, I think I can get that fixed up in the next day or two.

Oh, it’s PKCS8_pkey_set0 that failed?

It seems I misremembered the part of the if failed for something I checked only 2 hours ago 😄. You’re right. That also lines up with the error queue.

any chance we are trying to allocate negative number?

It’s not us allocating, it’s OpenSSL setting the error here:

https://github.com/openssl/openssl/blob/e5fb4b1469f317aa92768cdf804dfa29b72cb8f3/providers/implementations/encode_decode/encode_key2any.c#L94

    if ((p8info = PKCS8_PRIV_KEY_INFO_new()) == NULL
        || (derlen = k2d(key, &der)) <= 0
        || !PKCS8_pkey_set0(p8info, OBJ_nid2obj(key_nid), 0,
                            params_type, params, der, derlen)) {
        ERR_raise(ERR_LIB_PROV, ERR_R_MALLOC_FAILURE); // <-- Error is set here

I’m guessing the ERR_R_MALLOC_FAILURE makes sense if p8info is null, but it’s the PKCS8_pkey_set0 that fails and it lumps that in with the allocation failure.