runtime: Failure when building self-hosted NativeAOT compiler changes with 8.0 Preview 2 SDK

The changes in https://github.com/dotnet/runtime/pull/81205 have caused a failure when attempting to use .NET 8 Preview 2 SDK to build the Preview 3 source. This is a required scenario for .NET’s Source-Build. This was identified in the changes to update the source-build configuration so that it references the Preview 2 SDK: https://github.com/dotnet/installer/pull/15851.

The error that occurs is for the ILCompiler project:

EXEC : error : Failed to load assembly 'System.CommandLine' [/vmr/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj]
      Internal.TypeSystem.TypeSystemException+FileNotFoundException: Failed to load assembly 'System.CommandLine'
         at Internal.TypeSystem.ThrowHelper.ThrowFileNotFoundException(ExceptionStringID, String) in /_/src/coreclr/tools/Common/TypeSystem/Common/ThrowHelper.cs:line 35
         at Internal.TypeSystem.ResolutionFailure.Throw() in /_/src/coreclr/tools/Common/TypeSystem/Common/ResolutionFailure.cs:line 105
         at Internal.TypeSystem.Ecma.EcmaModule.GetObject(EntityHandle, NotFoundBehavior ) in /_/src/coreclr/tools/Common/TypeSystem/Ecma/EcmaModule.cs:line 412
         at Internal.TypeSystem.Ecma.EcmaModule.GetType(EntityHandle) in /_/src/coreclr/tools/Common/TypeSystem/Ecma/EcmaModule.cs:line 359
         at Internal.TypeSystem.Ecma.EcmaType.InitializeBaseType() in /_/src/coreclr/tools/Common/TypeSystem/Ecma/EcmaType.cs:line 163
         at Internal.TypeSystem.Ecma.EcmaType.ComputeTypeFlags(TypeFlags) in /_/src/coreclr/tools/Common/TypeSystem/Ecma/EcmaType.cs:line 199
         at Internal.TypeSystem.TypeDesc.InitializeTypeFlags(TypeFlags) in /_/src/coreclr/tools/Common/TypeSystem/Common/TypeDesc.cs:line 135
         at Internal.TypeSystem.Ecma.EcmaType.ComputeTypeFlags(TypeFlags) in /_/src/coreclr/tools/Common/TypeSystem/Ecma/EcmaType.cs:line 247
         at Internal.TypeSystem.TypeDesc.InitializeTypeFlags(TypeFlags) in /_/src/coreclr/tools/Common/TypeSystem/Common/TypeDesc.cs:line 135
         at Internal.TypeSystem.MetadataFieldLayoutAlgorithm.ComputeInstanceLayout(DefType, InstanceLayoutKind) in /_/src/coreclr/tools/Common/TypeSystem/Common/MetadataFieldLayoutAlgorithm.cs:line 41
         at Internal.TypeSystem.DefType.ComputeInstanceLayout(InstanceLayoutKind) in /_/src/coreclr/tools/Common/TypeSystem/Common/DefType.FieldLayout.cs:line 436
         at ILCompiler.CompilerTypeSystemContext.EnsureLoadableTypeUncached(TypeDesc) in /_/src/coreclr/tools/Common/Compiler/CompilerTypeSystemContext.Validation.cs:line 137
         at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey) in /_/src/coreclr/tools/Common/TypeSystem/Common/Utilities/LockFreeReaderHashtable.cs:line 562
         at ILCompiler.DependencyAnalysis.EETypeNode..ctor(NodeFactory, TypeDesc) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/EETypeNode.cs:line 85
         at ILCompiler.DependencyAnalysis.NodeFactory.CreateConstructedTypeNode(TypeDesc) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs:line 542
         at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey) in /_/src/coreclr/tools/Common/TypeSystem/Common/Utilities/LockFreeReaderHashtable.cs:line 562
         at ILCompiler.DependencyAnalysis.NodeFactory.ConstructedTypeSymbol(TypeDesc) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/NodeFactory.cs:line 614
         at ILCompiler.DependencyAnalysis.ReflectionInvokeMapNode.AddDependenciesDueToReflectability(DependencyList&, NodeFactory, MethodDesc) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectionInvokeMapNode.cs:line 54
         at ILCompiler.MetadataManager.GetDependenciesDueToReflectability(DependencyList&, NodeFactory, MethodDesc) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/MetadataManager.cs:line 399
         at ILCompiler.DependencyAnalysis.ReflectedMethodNode.GetStaticDependencies(NodeFactory) in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ReflectedMethodNode.cs:line 39
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependenciesImpl(DependencyNodeCore`1) in /_/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/DependencyAnalyzer.cs:line 182
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependencies(DependencyNodeCore`1) in /_/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/DependencyAnalyzer.cs:line 222
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ProcessMarkStack() in /_/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/DependencyAnalyzer.cs:line 257
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ComputeMarkedNodes() in /_/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/DependencyAnalyzer.cs:line 308
         at ILCompiler.ILScanner.ILCompiler.IILScanner.Scan() in /_/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/ILScanner.cs:line 140
         at ILCompiler.Program.<Run>g__RunScanner|3_0(<>c__DisplayClass3_0&) in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 438
         at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 418
         at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass203_0.<.ctor>b__0(InvocationContext) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 272

An analysis of the error and its symptoms is given here: https://github.com/dotnet/installer/pull/15851#issuecomment-1474447976. As stated in the analysis, the behavior is sporadic for some reason. One notable symptom is that BadImageFormatException is thrown when attempting to load certain assemblies in the ComputeManagedAssembliesToCompileToNative build task.

It’s been confirmed that reverting the changes in https://github.com/dotnet/runtime/pull/81205 resolves the issue. We should consider reverting this change or getting a fix in to address this in the Preview 3 timeframe.

cc @MichalStrehovsky

About this issue

Original URL
State: closed
Created a year ago
Comments: 34 (34 by maintainers)

Commits related to this issue

Exclude System.* reference assemblies in ILCompiler.Build.Tasks We expect the implementation of these assemblies to come as part of msbuild. Fixes #83695 — committed to jkotas/runtime by jkotas a year ago
Exclude System.* reference assemblies in ILCompiler.Build.Tasks (#86423) * Exclude System.* reference assemblies in ILCompiler.Build.Tasks We expect the implementation of these assemblies to come ... — committed to dotnet/runtime by jkotas a year ago
Apply source build patch Fixes #83695 — committed to jkotas/runtime by jkotas a year ago
Apply source build patch (#87226) Fixes #83695 — committed to dotnet/runtime by jkotas a year ago

Most upvoted comments

The failure is:

    EXEC : error : Failed to load assembly 'System.CommandLine' [/vmr/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj]
      Internal.TypeSystem.TypeSystemException+FileNotFoundException: Failed to load assembly 'System.CommandLine'
         at Internal.TypeSystem.ThrowHelper.ThrowFileNotFoundException(ExceptionStringID, String) + 0x30
         at Internal.TypeSystem.ResolutionFailure.Throw() + 0xfe
         at Internal.TypeSystem.Ecma.EcmaModule.GetObject(EntityHandle, NotFoundBehavior) + 0x9b
         at Internal.TypeSystem.Ecma.EcmaModule.GetType(EntityHandle) + 0x2c
         at Internal.TypeSystem.Ecma.EcmaType.InitializeBaseType() + 0x81
         at Internal.TypeSystem.Ecma.EcmaType.ComputeTypeFlags(TypeFlags) + 0x48
         at Internal.TypeSystem.TypeDesc.InitializeTypeFlags(TypeFlags) + 0x1e
         at Internal.TypeSystem.Ecma.EcmaType.ComputeTypeFlags(TypeFlags) + 0x1b4
         at Internal.TypeSystem.TypeDesc.InitializeTypeFlags(TypeFlags) + 0x1e
         at Internal.TypeSystem.MetadataFieldLayoutAlgorithm.ComputeInstanceLayout(DefType, InstanceLayoutKind) + 0x170
         at Internal.TypeSystem.DefType.ComputeInstanceLayout(InstanceLayoutKind) + 0x60
         at ILCompiler.CompilerTypeSystemContext.EnsureLoadableTypeUncached(TypeDesc) + 0x450
         at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey) + 0x14
         at ILCompiler.DependencyAnalysis.EETypeNode..ctor(NodeFactory, TypeDesc) + 0x14f
         at ILCompiler.DependencyAnalysis.NodeFactory.CreateConstructedTypeNode(TypeDesc) + 0x81
         at Internal.TypeSystem.LockFreeReaderHashtable`2.CreateValueAndEnsureValueIsInTable(TKey) + 0x14
         at ILCompiler.DependencyAnalysis.NodeFactory.ConstructedTypeSymbol(TypeDesc) + 0x77
         at ILCompiler.DependencyAnalysis.ReflectionInvokeMapNode.AddDependenciesDueToReflectability(DependencyNodeCore`1.DependencyList&, NodeFactory, MethodDesc) + 0x88
         at ILCompiler.MetadataManager.GetDependenciesDueToReflectability(DependencyNodeCore`1.DependencyList&, NodeFactory, MethodDesc) + 0x65
         at ILCompiler.DependencyAnalysis.ReflectedMethodNode.GetStaticDependencies(NodeFactory) + 0x6a
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependenciesImpl(DependencyNodeCore`1) + 0x35
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependencies(DependencyNodeCore`1) + 0x39
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ProcessMarkStack() + 0xb1
         at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ComputeMarkedNodes() + 0x54
         at ILCompiler.ILScanner.ILCompiler.IILScanner.Scan() + 0x19
         at ILCompiler.Program.<Run>g__RunScanner|3_0(Program.<>c__DisplayClass3_0&) + 0x117
         at ILCompiler.Program.Run() + 0x1637
         at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass206_0.<.ctor>b__0(InvocationContext context) + 0x201
    /vmr/src/runtime/artifacts/source-build/self/package-cache/microsoft.dotnet.ilcompiler/8.0.0-preview.4.23259.5/build/Microsoft.NETCore.Native.targets(270,5): error MSB3073: The command ""/vmr/src/runtime/artifacts/source-build/self/package-cache/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.0-preview.4.23259.5/tools/ilc" @"/vmr/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/ILCompiler/x64/Release/native/ilc.ilc.rsp"" exited with code 1. [/vmr/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj]

Notice that this is running .NET 8.0 Preview 4 build from package-cache (package-cache/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.0-preview.4.23259.5). This fix is present in .NET 8.0 Preview 5+ only. The failure should go away once this part of the source build is upgraded to .NET 8.0 Preview 5+.

jkotas on May 23, 2023

Yep, I have figured that based on Sven’s hint. Based on earlier comments, I assumed that this is a race condition that will reproduce intermittently, and I did not expect there to be a patch for it. It turns out that this is not a race condition. It reproduces 100% of time with the workaround patch removed.

jkotas on May 18, 2023

@sbomer Could you also look at this? This seems to be some incredibly strange race condition that results in a partially-written file being read by another task. From @MichalStrehovsky’s comment, it sounds like we might be able to just avoid some complexity here entirely by modifying the SDK.

agocke on May 16, 2023