azure-pipelines-agent: TFS Build Agent failing against TFS 2017 on premise TF400813: Resource not available for anonymous access. Client authentication required.

Agent Version of your agent: 2.105.7 OS of the machine running the agent: Windows TFS: On Prem, Version 2017

Issue: The build server has started to fail all builds with the error: Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: TF400813: Resource not available for anonymous access. Client authentication required.

I upgraded from TFS 2015.3 to TFS 2017, upgraded the build agent, and everything was working including builds. A day later the builds start to fail with the above error. I have been investigating and trying may different things to resolve the issue, but have had no luck. We have tried setting up another build server, and every server we configure it on, we always get the same error. We have gone as far as making our build user an administrator of TFS. The build user has administer permissions for the agent pool. Below is an excerpt of the work diag log, and the http fiddler trace.

Worker’s diag log

[2017-01-18 18:00:21Z INFO Worker] Listening for cancel message from the channel.
[2017-01-18 18:00:21Z INFO Worker] Waiting for the job to complete or for a cancel message from the channel.
[2017-01-18 18:00:22Z INFO Worker] Job completed.
[2017-01-18 18:00:22Z ERR  Program] Microsoft.VisualStudio.Services.Common.VssUnauthorizedException: TF400813: Resource not available for anonymous access. Client authentication required.
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.<SendAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.<SendAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.ThrottlingReportHandler.<SendAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__45.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__42`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.<GetConnectionDataAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.<ConnectAsync>d__41.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.JobServer.<ConnectAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.<RunAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Worker.<RunAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Worker.Program.<MainAsync>d__1.MoveNext()

Fiddler HTTP Trace I have traced the HTTP Messages All communication looks good until it attempts a request to a Project Collection api

Build Agent Request

GET https://tfs.*****.com/tfs/Sandbox/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 HTTP/1.1
Connection: Keep-Alive
Accept-Encoding: gzip
Accept-Language: en-US
Authorization: Bearer <Authroization Bearer Code>
Expect: 100-continue
User-Agent: VSServices/15.255.65000.0 (NetStandard) VstsAgentCore-win7-x64/2.105.7 (Microsoft Windows 10.0.14393)
X-TFS-FedAuthRedirect: Suppress
X-TFS-Session: d3e5a5ea-dec4-4c21-b186-b78496d19414
Host: tfs.*****.com

TFS Server Response:

HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/10.0
X-TFS-ProcessId: 96021cb3-66d9-48f2-898f-d5412ff210a6
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 3600
Access-Control-Allow-Methods: OPTIONS,GET,POST,PATCH,PUT,DELETE
Access-Control-Expose-Headers: ActivityId,X-TFS-Session,X-MS-ContinuationToken
Access-Control-Allow-Headers: authorization
X-FRAME-OPTIONS: SAMEORIGIN
X-TFS-SoapException: %3c%3fxml+version%3d%221.0%22+encoding%3d%22utf-8%22%3f%3e%3csoap%3aEnvelope+xmlns%3asoap%3d%22http%3a%2f%2fwww.w3.org%2f2003%2f05%2fsoap-envelope%22%3e%3csoap%3aBody%3e%3csoap%3aFault%3e%3csoap%3aCode%3e%3csoap%3aValue%3esoap%3aReceiver%3c%2fsoap%3aValue%3e%3csoap%3aSubcode%3e%3csoap%3aValue%3eUnauthorizedRequestException%3c%2fsoap%3aValue%3e%3c%2fsoap%3aSubcode%3e%3c%2fsoap%3aCode%3e%3csoap%3aReason%3e%3csoap%3aText+xml%3alang%3d%22en%22%3eTF400813%3a+Resource+not+available+for+anonymous+access.+Client+authentication+required.%3c%2fsoap%3aText%3e%3c%2fsoap%3aReason%3e%3c%2fsoap%3aFault%3e%3c%2fsoap%3aBody%3e%3c%2fsoap%3aEnvelope%3e
X-TFS-ServiceError: TF400813%3a+Resource+not+available+for+anonymous+access.+Client+authentication+required.
WWW-Authenticate: Bearer
WWW-Authenticate: Basic realm="https://tfs.*****.com/tfs"
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
Lfs-Authenticate: NTLM
X-Content-Type-Options: nosniff
Date: Wed, 18 Jan 2017 15:03:12 GMT
Content-Length: 19552