sdk: Multi-proc build of a .sln file is 'The process cannot access the file because it is being used by another process.'

In the CLI repo, we have a Test.sln file which we dotnet build. However, we are seeing random failures that look like the following:

05:25:35.797     1>CSC : error CS2012: Cannot open 'D:\j\workspace\debug_windows---54dcb9ec\test\Msbuild.Tests.Utilities\obj\Debug\netcoreapp1.0\Msbuild.Tests.Utilities.dll' for writing -- 'The process cannot access the file 'D:\j\workspace\debug_windows---54dcb9ec\test\Msbuild.Tests.Utilities\obj\Debug\netcoreapp1.0\Msbuild.Tests.Utilities.dll' because it is being used by another process.' [D:\j\workspace\debug_windows---54dcb9ec\test\Msbuild.Tests.Utilities\Msbuild.Tests.Utilities.csproj] [D:\j\workspace\debug_windows---54dcb9ec\build.proj]

See https://ci.dot.net/job/dotnet_cli/job/rel_1.0.0/job/debug_windows_nt_x86_prtest/810/console

To repro:

On Windows x64 machine, clone the CLI repo and checkout https://github.com/dotnet/cli/pull/5465/commits/f6b0196b0f8a7342d4b5fe5709568af9e3ede665.
Then remove the line https://github.com/dotnet/cli/pull/5465/commits/f6b0196b0f8a7342d4b5fe5709568af9e3ede665#diff-70eb062c7648e65aa7aff25e924d014bR84 which is working around this problem by turning off multi-proc build.
build.cmd

An easier repro once you have built one time is:

F:\cli>git clean test -xdf
F:\cli>.dotnet_stage0\x64\dotnet.exe restore test\Microsoft.DotNet.Cli.Tests.sln
F:\cli>.dotnet_stage0\x64\dotnet.exe build test\Microsoft.DotNet.Cli.Tests.sln

F:\cli>.dotnet_stage0\x64\dotnet.exe build test\Microsoft.DotNet.Cli.Tests.sln
Microsoft (R) Build Engine version 15.1.523.56541
Copyright (C) Microsoft Corporation. All rights reserved.

F:\cli\.dotnet_stage0\x64\sdk\2.0.0-alpha-004616\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.Sdk.DefaultItems.targets(101,5): warning : A PackageReference for 'NETStandard.Library' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\Microsoft.DotNet.Tools.Tests.Utilities.csproj]
F:\cli\.dotnet_stage0\x64\sdk\2.0.0-alpha-004616\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.Sdk.DefaultItems.targets(101,5): warning : A PackageReference for 'NETStandard.Library' was included in your project. This package is implicitly referenced by the .NET SDK and you do not typically need to reference it from your project. For more information, see https://aka.ms/sdkimplicitrefs [F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\Microsoft.DotNet.Tools.Tests.Utilities.csproj]
  ArgumentsReflector -> F:\cli\test\ArgumentsReflector\bin\Debug\netcoreapp2.0\ArgumentsReflector.dll
  Microsoft.DotNet.Cli.Utils -> F:\cli\src\Microsoft.DotNet.Cli.Utils\bin\Debug\netstandard1.5\Microsoft.DotNet.Cli.Utils.dll
  Microsoft.DotNet.TestFramework -> F:\cli\src\Microsoft.DotNet.TestFramework\bin\Debug\netstandard1.5\Microsoft.DotNet.TestFramework.dll
  Microsoft.DotNet.InternalAbstractions -> F:\cli\src\Microsoft.DotNet.InternalAbstractions\bin\Debug\netstandard1.3\Microsoft.DotNet.InternalAbstractions.dll
CSC : error CS2012: Cannot open 'F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.dll' for writing -- 'The process cannot access the file 'F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\obj\Debug\netstandard1.5\Microsoft.DotNet.Tools.Tests.Utilities.dll' because it is being used by another process.' [F:\cli\test\Microsoft.DotNet.Tools.Tests.Utilities\Microsoft.DotNet.Tools.Tests.Utilities.csproj]

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 2
Comments: 40 (37 by maintainers)

Commits related to this issue

Respect projects that say they only have one TFM Partially fixes dotnet/sdk#739. The first iteration of cross-targeting support code unconditionally queried each ProjectReference for the best TFM to... — committed to rainersigwald/msbuild by rainersigwald 7 years ago
Respect projects that say they only have one TFM Partially fixes dotnet/sdk#739. The first iteration of cross-targeting support code unconditionally queried each ProjectReference for the best TFM to... — committed to rainersigwald/sdk by rainersigwald 7 years ago
Respect projects that say they only have one TFM (#1667) Partially fixes dotnet/sdk#739. The first iteration of cross-targeting support code unconditionally queried each ProjectReference for the ... — committed to dotnet/msbuild by rainersigwald 7 years ago
Report that a project only has one TFM Partially fixes dotnet/sdk#739. The first iteration of cross-targeting support code unconditionally queried each ProjectReference for the best TFM to build aga... — committed to rainersigwald/sdk by rainersigwald 7 years ago
Update dependencies from https://github.com/aspnet/AspNetCore build 20190629.2 (#739) - Microsoft.AspNetCore.Mvc.Analyzers - 3.0.0-preview7.19329.2 - Microsoft.AspNetCore.Mvc.Api.Analyzers - 3.0.0-pr... — committed to mmitche/sdk by dotnet-maestro[bot] 5 years ago

Most upvoted comments

That’s a great repro, @AlgorithmsAreCool, thanks!

C:\src\AlgorithmsAreCool-sdk-739>dotnet publish -r win7-x86 /logger:C:\src\ParallelBuildDebuggingLogger\ParallelBuildDebuggingLogger\bin\Debug\netstandard1.4\ParallelBuildDebuggingLogger.dll;CoreCompile /m:1 | findstr /rc:"Building target.*HigherLib.csproj"
Building target 'CoreCompile' in {4: "C:\src\AlgorithmsAreCool-sdk-739\HigherLib\HigherLib.csproj" + <SolutionDir = C:\src\AlgorithmsAreCool-sdk-739\; BuildingSolutionFile = true; Configuration = Debug; Platform = AnyCPU; SolutionFileName = AlgorithmsAreCool-sdk-739.sln; CurrentSolutionConfigurationContents = <SolutionConfiguration>
Building target 'CoreCompile' in {10: "C:\src\AlgorithmsAreCool-sdk-739\HigherLib\HigherLib.csproj" + <TargetFramework = netstandard1.6>}

This shows that the race is between

The solution-driven invocation of HigherLib
- New set of global properties is common to all builds invoked in the solution and contains solution metadata
- Because this is a single-TFM project, this implicitly is the “inner” build, which does I/O work
The ProjectReference-driven invocation of HigherLib for TargetFramework = netstandard1.6
- New set of global properties is the explicit TFM desired for compatibility with App
- Explicit request for the inner build

That explains a few questions I had (“why only in solutions?” and “why can’t I repro this with my multitargeting attempts?”).

@nguerrera @natidea I think we can fix this without going all the way to fixing https://github.com/Microsoft/msbuild/issues/1276 (which would probably be ideal but is invasive). Would it be reasonable to respond to the GetTargetFrameworkProperties request with no information if the project has a single TFM (just as we do for legacy projects)? That would collapse the latter request into the former and the engine would just return the previously-built file.

rainersigwald on Jan 31, 2017

No, the rule is that the engine builds only once for each tuple of:

Project
Target
Set of global properties

It’s definitely concerning that something in the build is building project 4 22 times.

I’m going to apply some of my magic internal tracing tools to the Sdk repo to help chase this down.

rainersigwald on Jan 30, 2017