runtime: Windows ARM64 build gets stuck at installer.tasks

I’ve confirmed that the incoming SDK version update from @ViktorHofer (#41947) solves this particular problem.

The root cause of the hang is https://github.com/dotnet/runtime/issues/39701

It’s rearing its ugly head at this callstack:

        Child SP               IP Call Site
000000DB4757EB40 00007ff9e6eb39ec [PrestubMethodFrame: 000000db4757eb40] System.Runtime.CompilerServices.DependentHandle.nSetPrimary(IntPtr, System.Object)
000000DB4757ED40 00007ff9aea62124 System.Runtime.CompilerServices.ConditionalWeakTable`2+Container[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].Remove(System.__Canon) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/ConditionalWeakTable.cs @ 577]
000000DB4757ED70 00007ff9aea60f18 System.Runtime.CompilerServices.ConditionalWeakTable`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].Remove(System.__Canon)
000000DB4757EDB0 00007ff9aeb8b0ac System.Collections.Generic.Dictionary`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].OnDeserialization(System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Collections/Generic/Dictionary.cs @ 687]
000000DB4757EE00 00007ff9b2b99c18 System.Runtime.Serialization.ObjectManager.RaiseDeserializationEvent() [/_/src/libraries/System.Runtime.Serialization.Formatters/src/System/Runtime/Serialization/ObjectManager.cs @ 971]
000000DB4757EE20 00007ff9b2ba9908 System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(System.Runtime.Serialization.Formatters.Binary.BinaryParser, Boolean) [/_/src/libraries/System.Runtime.Serialization.Formatters/src/System/Runtime/Serialization/Formatters/Binary/BinaryObjectReader.cs @ 128]
000000DB4757EE70 00007ff9b2ba31ac System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(System.IO.Stream, Boolean) [/_/src/libraries/System.Runtime.Serialization.Formatters/src/System/Runtime/Serialization/Formatters/Binary/BinaryFormatter.cs @ 69]
000000DB4757EEB0 00007ff9b2ba2ffc System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(System.IO.Stream) [/_/src/libraries/System.Runtime.Serialization.Formatters/src/System/Runtime/Serialization/Formatters/Binary/BinaryFormatter.cs @ 41]
000000DB4757EEC0 00007ff955bcd60c Microsoft.Build.BackEnd.BinaryTranslator+BinaryReadTranslator.TranslateDotNet[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef) [C:\src\msbuild\src\Shared\BinaryTranslator.cs @ 416]
000000DB4757EF20 00007ff955bcab70 Microsoft.Build.Shared.LogMessagePacketBase.ReadFromStream(Microsoft.Build.BackEnd.ITranslator) [C:\src\msbuild\src\Shared\LogMessagePacketBase.cs @ 400]
000000DB4757F000 00007ff955bc96ac Microsoft.Build.Shared.LogMessagePacketBase.Translate(Microsoft.Build.BackEnd.ITranslator) [C:\src\msbuild\src\Shared\LogMessagePacketBase.cs @ 261]
000000DB4757F030 00007ff955bc95e0 Microsoft.Build.Shared.LogMessagePacketBase..ctor(Microsoft.Build.BackEnd.ITranslator) [C:\src\msbuild\src\Shared\LogMessagePacketBase.cs @ 192]
000000DB4757F050 00007ff955bc957c Microsoft.Build.BackEnd.LogMessagePacket..ctor(Microsoft.Build.BackEnd.ITranslator) [C:\src\msbuild\src\Build\BackEnd\Components\Communications\LogMessagePacket.cs @ 36]
000000DB4757F070 00007ff955bc9538 Microsoft.Build.BackEnd.LogMessagePacket.FactoryForDeserialization(Microsoft.Build.BackEnd.ITranslator) [C:\src\msbuild\src\Build\BackEnd\Components\Communications\LogMessagePacket.cs @ 45]
000000DB4757F090 00007ff955bc94d0 Microsoft.Build.BackEnd.NodePacketFactory+PacketFactoryRecord.DeserializeAndRoutePacket(Int32, Microsoft.Build.BackEnd.ITranslator) [C:\src\msbuild\src\Shared\NodePacketFactory.cs @ 104]
000000DB4757F0D0 00007ff955bc93c0 Microsoft.Build.BackEnd.NodePacketFactory.DeserializeAndRoutePacket(Int32, Microsoft.Build.BackEnd.NodePacketType, Microsoft.Build.BackEnd.ITranslator) [C:\src\msbuild\src\Shared\NodePacketFactory.cs @ 60]
000000DB4757F110 00007ff955bc92ac Microsoft.Build.BackEnd.NodeManager.DeserializeAndRoutePacket(Int32, Microsoft.Build.BackEnd.NodePacketType, Microsoft.Build.BackEnd.ITranslator) [C:\src\msbuild\src\Build\BackEnd\Components\Communications\NodeManager.cs @ 277]
000000DB4757F140 00007ff955bc7d68 Microsoft.Build.BackEnd.NodeProviderOutOfProcBase+NodeContext.ReadAndRoutePacket(Microsoft.Build.BackEnd.NodePacketType, Byte[], Int32) [C:\src\msbuild\src\Build\BackEnd\Components\Communications\NodeProviderOutOfProcBase.cs @ 928]
000000DB4757F1D0 00007ff955b8b3d8 Microsoft.Build.BackEnd.NodeProviderOutOfProcBase+NodeContext+d__12.MoveNext() [C:\src\msbuild\src\Build\BackEnd\Components\Communications\NodeProviderOutOfProcBase.cs @ 719]

The exception goes unhandled and the out-of-proc packet processing loop breaks.

Until the underlying FCall issue is fixed, please try setting the MSBuildUseLegacyStringInterner environment variable to 1. This disables the code that calls GCHandle.set_Target, which should work around the collision.

Here’s my repro as requested by @safern:

1. My laptop is Surface Pro X, with Windows 10 Version 1909 (Build 18363.900).
2. Installed PowerShell Core ARM64 - No installer, need to download the zip from here and extract it in “C:\Program Files (Arm)\PowerShell\7-preview”.
3. Installed VS and all requirements, CMake (note there’s no Arm64 version, not sure if relevant), and cloned runtime.
4. Installed dotnet via the dotnet-install script, using the version required by the runtime repo, double checked that the environment variable got set to point to the install location:

.\dotnet-install.ps1 -Architecture arm64 -InstallDir "C:\Program Files (Arm)\dotnet" -JsonFile C:\Users\myusername\source\repos\runtime\global.json 

1. Building runtime gets stuck in an “installer.tasks” step:

PS C:\Users\myusername\source\repos\runtime> .\build.cmd clr+libs -rc release -a arm64
  Determining projects to restore...
  Tool 'coverlet.console' (version '1.7.2') was restored. Available commands: coverlet
  Tool 'dotnet-reportgenerator-globaltool' (version '4.5.8') was restored. Available commands: reportgenerator
  Tool 'microsoft.dotnet.xharness.cli' (version '1.0.0-prerelease.20329.4') was restored. Available commands: xharness

  Restore was successful.
  Restored C:\Users\myusername\.nuget\packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20316.1\tools\Tools.proj (in 419 ms).
  Determining projects to restore...
  Restored C:\Users\myusername\source\repos\runtime\tools-local\tasks\tasks.proj (in 262 ms).
  Restored C:\Users\myusername\source\repos\runtime\tools-local\tasks\installer.tasks\installer.tasks.csproj (in 2.52 sec).
  installer.tasks -> C:\Users\myusername\source\repos\runtime\artifacts\bin\installer.tasks\Debug\netstandard2.0\installer.tasks.dll

1. Attempting to build installer.tasks.csproj by itself with diagnostics logging gets stuck after these messages, and freezes PowerShell (can’t kill the process). I uploaded the full binlog file in OneDrive (it’s ~5MB). If you’d like to see it, please contact me via Teams so I can give you the link.

PS C:\Users\myusername\source\repos\runtime> dotnet build /v:diag .\tools-local\tasks\installer.tasks\installer.tasks.csproj
...
13:01:07.920   1:2>Project "C:\Users\myusername\source\repos\runtime\tools-local\tasks\installer.tasks\installer.tasks.csproj" (1:2) is building "C:\Users\myusername\source\repos\runtime\tools-local\tasks\installer.tasks\installer.tasks.csproj" (1:4) on node 2 (Build target(s)).
13:01:08.878   1:4>Building with tools version "Current".
                   Target "ErrorForMissingTestRunner" skipped, due to false condition; ('$(IsTestProject)' == 'true' AND '$(TestRunnerName)' != '') was evaluated as ('false' == 'true' AND '' != '').
                   Target "_CheckForUnsupportedTargetFramework" skipped, due to false condition; ('$(_UnsupportedTargetFrameworkError)' == 'true') was evaluated as ('' == 'true').
13:01:08.888   1:4>Target "_CollectTargetFrameworkForTelemetry: (TargetId:2)" in file "C:\Program Files (Arm)\dotnet\sdk\5.0.100-preview.6.20310.4\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.TargetFrameworkInference.targets" from project "C:\Users\myusername\source\repos\runtime\tools-local\tasks\installer.tasks\installer.tasks.csproj" (target "_CheckForInvalidConfigurationAndPlatform" depends on it):
                   Using "AllowEmptyTelemetry" task from assembly "C:\Program Files (Arm)\dotnet\sdk\5.0.100-preview.6.20310.4\Sdks\Microsoft.NET.Sdk\targets\..\tools\net5.0/Microsoft.NET.Build.Tasks.dll".
                   Task "AllowEmptyTelemetry" (TaskId:2)
                     Task Parameter:EventName=targetframeworkeval (TaskId:2)
                     Task Parameter:EventData=TargetFrameworkVersion=.NETFramework,Version=v4.6;RuntimeIdentifier=;SelfContained=;UseApphost=;OutputType=Library (TaskId:2)

@eiriktsarpalis I think we can reduce the log and make it smaller if you do:

dotnet build tools-local\tasks\installer.tasks\ /p:TargetArchitecture=arm64 /v:diag

Here’s my output using build.cmd clr+libs -a arm64 -rc release -v diag

log.txt