sdk: Build fails due to locked file which it locks itself (concurrent build problem)

I second the vote for “make it work somehow”. Telling people they have to manually build every project separately is ridiculous. People expect to just build a solution and for the build engine to work.

From that log:

547257:Project "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-CD106-LIN\Tic.sln" (1) is building "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-CD106-LIN\test\Integration.Kardex\Integration.Kardex.csproj" (35) on node 1 (Publish target(s)).
637706:Project "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-CD106-LIN\test\Integration.Kardex.Wamp.AspNetCore\Integration.Kardex.Wamp.AspNetCore.csproj" (37) is building "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-CD106-LIN\test\Integration.Kardex\Integration.Kardex.csproj" (35:2) on node 5 (GetTargetFrameworks target(s)).
3075143:Project "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-CD106-LIN\test\Integration.Kardex.Wamp.AspNetCore\Integration.Kardex.Wamp.AspNetCore.csproj" (37) is building "D:\Atlassian\Bamboo\xml-data\build-dir\TIC-CD106-LIN\test\Integration.Kardex\Integration.Kardex.csproj" (35:3) on node 2 (default targets).

That indicates that Integration.Kardex.csproj is building with 3 separate sets of global properties. The call to GetTargetFrameworks should be harmless–it should only return information about the project, not do any actual work with files on disk. That leaves the solution reference and the ProjectReference from Integration.Kardex.Wamp.AspNetCore.csproj.

As shown in that image, both of the racing instances of the project actually have the same TF and RID, which is where I expected to see the global property change. Unfortunately, the binary log wasn’t captured with an MSBuild that has https://github.com/Microsoft/msbuild/pull/3252 (fixed in 15.8 that’s not yet released), so it’s pretty hard to figure out what properties are different.

Based on testing on a simplified project (one mvc project referencing one netcoreapp2.1 classlib), I think there’s a race between the project as built by the solution (inheriting the RID and TF passed to the solution by dotnet publish) and the project as built by the reference (with explicitly unset RID and TF, because there’s only one in the file).

But that doesn’t seem to be what’s happening here, because the query-TFs call didn’t collapse into the build-single-TF call. I can’t tell why from the logging; I’m getting lost in the various different properties set in different projects.

There’s definitely a problem here. At the moment I would suggest running publish only on individual projects and not at the solution level. That will help ensure that TargetFramework and RID flow correctly down to referenced projects.

My workaround for this is to disable concurrent build like this:

dotnet publish -maxcpucount:1 <rest of args>

But a proper fix would certainly be nice 😃

As a sidenode: @rainersigwald mentioned that references were built multiple times. We had the exact same issue and found that one of out net core project files had the TargetFramework tag instead of TargetFrameworks. Once we changed it in the .csproj to TargetFrameworks- even though it was only one target framework- the extra builds of the same project and the resulting errors were gone.

(I continue to be amazed by the work @rainersigwald puts into explaining build issues. now its full-blown diagrams, I still remember some hand-drawn diagrams about RID conflicts ^^)

Any updates on this ?

So, i can repro this parallel build issue extremely simple, and publish is not needed, but dotnet restore is enough:

Reproduction:

• Create a new solution
• Add an ASP.NET core App
• Add a .NET Standrd 2.0 library
• Run dotnet restore TheNameYouChose.sln
• run psh> Get-Process -Name "dotnet"
• See leftover dotnet process

Probably works with different setups equally easy. Individual csproj builds work just fine, of course

λ  dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.2.402
 Commit:    c7f2f96116

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18362
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.402\

Host (useful for support):
  Version: 3.0.0-preview8-28405-07
  Commit:  d01b2fb7bc

.NET Core SDKs installed:
  2.1.500 [C:\Program Files\dotnet\sdk]
  2.1.801 [C:\Program Files\dotnet\sdk]
  2.2.402 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview8.19405.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview8-28405-07 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview8-28405-07 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

So… Is there a solution to this? It disrupts our automated build processes…

maybe @rainersigwald can help here.

In the meantime, it would really help if you’d be able to get a build log (-bl parameter will create an msbuild.binlog file). This file will also contain the contents of your project files (.csproj - not(!) source code / file contents) so you may want to only mail it to Microsoft employees or trustworthy individuals who are willing to help if these contain any secret infos. Do you have VS / any other form of tooling open during the publish operation? (haven’t heard this was a problem but just checking)

Has anything ever come out of this? I have a group of about 200 assemblies. Running DOTNET on each project is WAY too slow so I put them into a solution and it takes minutes (as opposed to hours individually). But I get a lot of these Copy/retries and sometimes they run out of retries and the build fails.

If I could pass in a list of projects (like when building the solution) that may be helpful, even if have to list them in build order. Possibly tell it to ignore locked files (because these files packages ARE there),

How are you building the solution? Are you passing any command line arguments?

I ask because issues like this can be caused by things like specifying the output path on the command line, and then multiple projects try to build to the same output path and can get file conflicts.

@smaktacular’s reccomendation works to fix the build of the actual project, but (for me) breaks the build of the related Docker/docker-compose.

Error for the project containing the Dockerfile:

error MSB4057: The target “DockerResolveAppType” does not exist in the project.

Workaround Change all projects in the solution to TargetFrameworks except for the one containing the Dockerfile, which should retain TargetFramework

I believe the problem with slns was first clearly identified with those famous diagrams. 😃 Yes, we should warn. In 3.0, we should either disallow it or actually make it work somehow.