runtime: Test failure: Manual_CertificateSentMatchesCertificateReceived_Success

Currently running the tests with the fix for dotnet/corefx#16516 to see if the test results change. Will report back when I have the information.

Thanks @steveharter ! These are all different issues from the original (which was a server-side crash or other kind of timeout). Do you think we should track using separate issues or keep the same since they all happen within this particular test?

The stacks are almost impossible to decipher - they require disassembly to understand what __DisplayClass3_0. ... b__0() and similar actually mean. On top of that, if the code changes with as much as adding another await all of the IDs change to mean something different… Even after doing so I found out that the actual error/exception was long gone and what we’re seeing is just the error state being marshaled to the test thread.

@karelz @davidsh @DavidGoll Based on the triage decision of time-boxing this investigation, I’m moving this to “Future”.

@steveharter proved that under stress, the HttpClient/WinHttpHandler has at least 3 failure modes. To continue the investigation my ideas would be to:

1. Use sufficient VMs that continuously run the tests in a tight loop. This increases the likelihood of repro-ing the problems as well as providing an A/B testing environment.
2. Instrument the code to Debug.Fail and dump memory on unknown errors such as ERROR_WINHTTP_CLIENT_CERT_NO_ACCESS_PRIVATE_KEY, ERROR_WINHTTP_INVALID_SERVER_RESPONSE, ERROR_WINHTTP_INCORRECT_HANDLE_TYPE, ERROR_WINHTTP_INCORRECT_HANDLE_STATE.
3. For errors such as Authentication failed because the remote party has closed the transport stream. we would also need to capture a full-memory dump and understand the state for the LoopbackServer/SslStream/NetworkStream to understand where they crash.
4. Fix the TestHelper.WhenAllCompletedOrAnyFailed pattern to ensure we somehow get sufficient state information from all other tasks when a test failure is logged.
5. Instrument LoopbackServer to fail fast (with a memory dump in order to see the client task state) and otherwise log each operation.
6. As a last resort, separate the test into two processes : client and server. Running on 2 separate machines this method will also allow capturing network traffic.

Any pointers on how to interpret the stack trace

The stack is:

System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.Functional.Tests.HttpClientHandler_ClientCertificates_Test.<>c__DisplayClass6_1.<<Manual_CertificateSentMatchesCertificateReceived_Success>b__2>d.MoveNext() in S:\corefx\src\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.ClientCertificates.cs:line 106
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.Functional.Tests.HttpClientHandler_ClientCertificates_Test.<>c__DisplayClass6_2.<<Manual_CertificateSentMatchesCertificateReceived_Success>b__5>d.MoveNext() in S:\corefx\src\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.ClientCertificates.cs:line 130
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Mentally delete these three lines from each section:

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

Those are infrastructure involved in propagating the exception stored in the task, and unfortunately don’t have a good way to hide it from stack traces.

That leaves you with:

System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.Functional.Tests.HttpClientHandler_ClientCertificates_Test.<>c__DisplayClass6_1.<<Manual_CertificateSentMatchesCertificateReceived_Success>b__2>d.MoveNext() in S:\corefx\src\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.ClientCertificates.cs:line 106
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.Functional.Tests.HttpClientHandler_ClientCertificates_Test.<>c__DisplayClass6_2.<<Manual_CertificateSentMatchesCertificateReceived_Success>b__5>d.MoveNext() in S:\corefx\src\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.ClientCertificates.cs:line 130
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

The async part of this should be pretty easy to understand: those two lines 106 and 130 are in the async method Manual_CertificateSentMatchesCertificateReceived_Success. The b__2 and b__5 actually have little to do with async/await: those are the names of the methods generated for the lambdas in the code (you would see similar names in regular synchronous code using lambdas).