azure-pipelines-agent: Multiple agents on the same machine using tf.exe can lead to getting sources failure

A build is intermittently getting this error when fetching source from TFS:

2017-03-23T23:49:31.0591599Z ##[section]Starting: Build solution Roster.WebService.sln
2017-03-23T23:49:31.0591599Z ==============================================================================
2017-03-23T23:49:31.0591599Z Task         : Visual Studio Build
2017-03-23T23:49:31.0591599Z Description  : Build with MSBuild and set the Visual Studio version property
2017-03-23T23:49:31.0591599Z Version      : 1.113.0
2017-03-23T23:49:31.0591599Z Author       : Microsoft Corporation
2017-03-23T23:49:31.0591599Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613727)
2017-03-23T23:49:31.0591599Z ==============================================================================
2017-03-23T23:49:31.6529233Z Unable to determine the workspace. You may be able to correct this by running 'tf workspaces /collection:TeamProjectCollectionUrl'.
2017-03-23T23:49:31.8872843Z ##[error]Exit code 100 returned from process: file name 'tf', arguments 'vc resolvePath "$\My Development\Trunk\src\Rostering\trunk\Roster.WebService.sln" /loginType:OAuth /login:.,******** /noprompt'.
2017-03-23T23:49:31.8872843Z ##[section]Finishing: Build solution Roster.WebService.sln

Copied from https://github.com/Microsoft/vsts-tasks/issues/3853

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 84 (36 by maintainers)

Most upvoted comments

Work is in progress to get a new agent that carries the updated tf.exe and dependency DLLs (pr https://github.com/Microsoft/vsts-agent/pull/1056)

ericsciple on Jul 5, 2017

On version 123 I am still having the issue, but I can add some details and a workaround that may be helpful to find this problem and get people back up and running again.

So here is the scenario as I have pinned down.

Remove all agents
Remove all workspaces with tf workspaces /remove:*
Run a build, get the error
Run Developer command Prompt
Navigate to folder inside of workspace
Run tf workfold > get same error 8a. Run tf workspaces /s:serverurl > a workspace shows up 8b. Alternative to 8a, open visual studio and open manage workspaces, this makes tf workfold show details as well. I think it calls the same workspaces call when it opens.
Run tf workfold > now the workspace resolves.
From this point the builds will work when they run in that workspace, unless the build has clean enabled, in which will delete the workspace then add it back with the same issue.

I too though it was random until I tracked the issue. Here is what mine did.

Build1 ran on Agent1 and failed for error, I went to build server and looked at the workspaces which fixed that build.
Build2 ran on Agent1 with same problem and I went to VS and opened workspaces again
at this point build1 and build2 had “Fixed” workspaces on that agent.
Now Build1 and Build2 run find as they always get queued on agent1.
Run Build1 and Build2 in Parrallel, now agent1 does build1 and agent2 takes build2 at which time build2 now gets the error.
Opening workspaces will fix Agent2s workspace for build2, or if you run again after build1 finishes it will probably start on Agent1 again which was "Fixed’ before.

So my assumption here is the workspace is being created but not attached to something. I think it’s the cache based on MSDN Blog
which states “If the workspace mapping exists on the server, then your mappings cache file may need to be refreshed. Try running this command to populate/refresh the cache: tf workspaces /s:http://tfs-server:8080” Which is why I tried 8a mentioned earlier and is the workaround if implemented as below.

WORKAROUND:

In the powershell call “tf workspaces /s:http://tfs-server.com” before calling tf whatever.

The only other current “Workaround” we have is,

Disable clean workspace in all build definitions.
Every time a new workspace is created (because a build runs on an agent it has not run on yet), manually run what is mentioned in 8a. From there you have to do this for every workspace, which in my case is 42 worst case, which is not feasible.

Here are the details for the questions asked.

Have you updated all your agents to 2.120.1? on 123
What version of VS is installed on the agents? 2015.2
Are all agents at the same version, with the same VS version? Not sure how the versions relate. Looks like agents are using ddls for 15 and VS is 14, so I assume they are different. But VS15 compared to 2.123 do not seem to relate well.
Build step details: How many agents are you running on the same machine? How often do these agents run in parallel? 6 agents 7 builds, 27 releases, so 204 workspaces max. Run parallel a lot.
If this is a regression, what was the last working version #? from old agents 14.102…
Latest build logs that carry the error supplied. I would have to go through all of the files and details to not violate our strict NDA and security agreements. But needed info is: VERBOSE: setting location to: C:\Builds\Agent1_work\1\s\mycode VERBOSE: exe: C:\Program Files (x86)\Microsoft Visual Studio 14.0\common7\ide\tf.exe VERBOSE: args: checkout SolutionInfo.cs Calling Cmd: C:\Program Files (x86)\Microsoft Visual Studio 14.0\common7\ide\tf.exe checkout SolutionInfo.cs VERBOSE: stdout: stderr: Unable to determine the workspace. You may be able to correct this by running ‘tf workspaces /collection:TeamProjectCollectionUrl’. VERBOSE: exit code 100

Now I am not sure if the issue is caused by having VS2015 installed while using agents for 2017, looking at the prior messages, it looks like everyone having this issue is using this scenario, so that may be the root problem. Several people have said they have VS2015, and I am running tf from version 14 (which is VS2015, so confusing). That may be the key to replicate the issue.

This is my setup for tf.exe $tfsTool = “C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\TF.exe” so directly using 14.

Sorry for the long message, but I hope this is detailed enough and contains enough information to fix or workaround the problem. Personally I think if this is an issue using 2015 TF and 2017 agents, then there may need to be a workaround posted and not necessarily a fix, but a fix would be nice.

metaphysico on Sep 19, 2017

@Cristie - this issue is only tracking the error mentioned at the beginning of this issue

From a build, get sources with multiple agents can cause:

Unable to determine the workspace. You may be able to correct this by running ‘tf workspaces /collection:TeamProjectCollectionUrl’. 2017-03-23T23:49:31.8872843Z ##[error]Exit code 100 returned from process

That is an issue in tf.exe shipped with the agent

bryanmacfarlane on Aug 30, 2017

those are separate issues. let’s keep this thread focused.

ericsciple on Aug 29, 2017

@JamesNK I’m trying to think of some workarounds…

Assuming you run the agents as a Windows service (and not interactively from command prompt), one thing that could mitigate is running the agents as different service accounts. My best guess right now is that some race condition is stomping on some local file.

If you don’t have complex mappings, another thing you could try is specify the file name as a relative path. The agent will prepend the build sources directory (e.g. C:\Agent1\_work\1\s). Or use $(Build.SourcesDirectory)/relative/path/to/file.ext.

Also I’m planning to add support to opt-in to server workspaces, by setting a variable on the definition (something like Build.TfvcWorkspaceType=Server). I can add that to the next agent release if you want to try it, and see if the issue goes away. My thinking is XAML build supported side-by-side agents and this issue afaik wasn’t a problem.

I’ll talk with VC folks again to see if they have ideas how to get to the bottom of this issue.

ericsciple on Apr 21, 2017