runtime: GSS failures in System.Net.Http.Functional.Tests on Ubuntu 22.04

Description

Running runtime tests on Ubuntu 22.04 (which adds OpenSSL 3.0 resulting in a number of changes under the hood), leads to a bunch of tests failing: https://dev.azure.com/dnceng/public/_build/results?buildId=1690650&view=ms.vss-test-web.build-test-results-tab&runId=46193442&resultId=189361&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab

Some examples:

System.AggregateException : One or more errors occurred. (GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).) (Unexpected EOF trying to read request header)
---- System.ComponentModel.Win32Exception : GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).
---- System.IO.IOException : Unexpected EOF trying to read request header
    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(authScheme: "NTLM") [FAIL]
      System.AggregateException : One or more errors occurred. (GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).) (Unexpected EOF trying to read request header)
      ---- System.ComponentModel.Win32Exception : GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).
      ---- System.IO.IOException : Unexpected EOF trying to read request header
      Stack Trace:
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(88,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(101,0): at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(669,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(String authScheme)
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace #1 (System.ComponentModel.Win32Exception) -----
        /_/src/libraries/Common/src/System/Net/Security/NegotiateStreamPal.Unix.cs(537,0): at System.Net.Security.NegotiateStreamPal.AcquireCredentialsHandle(String package, Boolean isServer, NetworkCredential credential)
        /_/src/libraries/Common/src/System/Net/NTAuthentication.Common.cs(128,0): at System.Net.NTAuthentication.Initialize(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding)
        /_/src/libraries/Common/src/System/Net/NTAuthentication.Common.cs(98,0): at System.Net.NTAuthentication..ctor(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs(169,0): at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(1033,0): at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.cs(240,0): at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs(30,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(532,0): at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(676,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.<>c__DisplayClass34_0.<<Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)
        ----- Inner Stack Trace #2 (System.IO.IOException) -----
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(715,0): at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestHeaderBytesAsync()
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(777,0): at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestDataAsync(Boolean readBody)
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(1021,0): at System.Net.Test.Common.LoopbackServer.Connection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(1081,0): at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(686,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.<>c__DisplayClass34_0.<<Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader>b__1>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)

Reproduction Steps

From helix:

/root/helix/work/correlation/dotnet exec --runtimeconfig System.Net.Http.Functional.Tests.runtimeconfig.json --depsfile System.Net.Http.Functional.Tests.deps.json xunit.console.dll System.Net.Http.Functional.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 

Expected behavior

All tests pass

Actual behavior

Tests fail with GSS exceptions.

Regression?

Yes, the tests pass on older versions of Ubuntu currently running in CI

Known Workarounds

No response

Configuration

  • dotnet/runtime main branch, commit 5a0564b01442f8ea9247e27c4fab85ee0d457265
  • Ubuntu 22.04

Other information

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 2
  • Comments: 19 (18 by maintainers)

Most upvoted comments

One way to tell is to change openssl.cnf to have the following provider_sect:

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

One way to tell is to change openssl.cnf to have the following provider_sect:

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

This helped. Thanks! 😃

I’m not sure. This looks like distribution bug to me @omajid as the package they provide does not work.