runtime: Test failure JIT/Methodical/xxobj/sizeof/sizeof64_Target_64Bit_and_arm_il_r/sizeof64_Target_64Bit_and_arm.cmd

Failed in run: runtime-coreclr outerloop 20230123.1

Failed tests:

R2R-CG2 windows arm Checked @ Windows.11.Arm64.Open
  - JIT/Methodical/xxobj/sizeof/sizeof64_Target_64Bit_and_arm_il_r/sizeof64_Target_64Bit_and_arm.cmd

Error message:

Fatal error. Internal CLR error. (0x80131506)
at System.GC.AllocateNewArray(IntPtr, Int32, GC_ALLOC_FLAGS)
at System.Text.StringBuilder.ExpandByABlock(Int32)
at System.Text.StringBuilder.Append(Char, Int32)
at System.Text.StringBuilder.Append(Char)
at System.Diagnostics.StackTrace.ToString(TraceFormat, System.Text.StringBuilder)
at System.Diagnostics.StackTrace.ToString(TraceFormat)
at System.Exception.get_StackTrace()
at System.Exception.ToString()
at Internal.JitInterface.CorInfoImpl.AllocException(System.Exception)
at Internal.JitInterface.CorInfoImpl._resolveToken(IntPtr, IntPtr*, Internal.JitInterface.CORINFO_RESOLVED_TOKEN*)
at Internal.JitInterface.CorInfoImpl.JitCompileMethod(IntPtr ByRef, IntPtr, IntPtr, IntPtr, Internal.JitInterface.CORINFO_METHOD_INFO ByRef, UInt32, IntPtr ByRef, UInt32 ByRef)
at Internal.JitInterface.CorInfoImpl.CompileMethodInternal(ILCompiler.DependencyAnalysis.IMethodNode, Internal.IL.MethodIL)
at Internal.JitInterface.CorInfoImpl.CompileMethod(ILCompiler.DependencyAnalysis.ReadyToRun.MethodWithGCInfo, ILCompiler.Logger)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOneMethod|5(ILCompiler.DependencyAnalysisFramework.DependencyNodeCore`1<ILCompiler.DependencyAnalysis.NodeFactory>, Int32)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompileOnThread|4(Int32)
at ILCompiler.ReadyToRunCodegenCompilation+<>c__DisplayClass46_0.<ComputeDependencyNodeDependencies>g__CompilationThread|3(System.Object)
at System.Threading.Thread.StartCallback()

Return code:      1
Raw output file:      C:hwA74E094BwB27F0950uploads\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_routput.txt
Raw output:
BEGIN EXECUTION
sizeof64_Target_64Bit_and_arm.dll
TestLibrary.dll
2 file(s) copied.
11:39:15.16
Response file: C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll.rsp
C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rIL-CG2sizeof64_Target_64Bit_and_arm.dll
-o:C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll
--targetarch:arm
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwA74E094BpSystem..dll
-r:C:hwA74E094BpMicrosoft..dll
-r:C:hwA74E094Bpmscorlib.dll
-r:C:hwA74E094Bp
etstandard.dll
-O
" "dotnet" "C:hwA74E094Bpcrossgen2crossgen2.dll" @"C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll.rsp"   -r:C:hwA74E094BwB27F0950eJITMethodicalMethodical_r2IL-CG2*.dll  -r:C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rIL-CG2*.dll"
Emitting R2R PE file: C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll
" "dotnet" "C:hwA74E094Bp
2rdump
2rdump.dll" --header --sc --in C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll --out C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rsizeof64_Target_64Bit_and_arm.dll.r2rdump --val"
11:39:16.88
11:39:16.90
Response file: C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rTestLibrary.dll.rsp
C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rIL-CG2TestLibrary.dll
-o:C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rTestLibrary.dll
--targetarch:arm
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwA74E094BpSystem..dll
-r:C:hwA74E094BpMicrosoft..dll
-r:C:hwA74E094Bpmscorlib.dll
-r:C:hwA74E094Bp
etstandard.dll
-O
" "dotnet" "C:hwA74E094Bpcrossgen2crossgen2.dll" @"C:hwA74E094BwB27F0950eJITMethodical\xxobjsizeofsizeof64_Target_64Bit_and_arm_il_rTestLibrary.dll.rsp"   -r:C:hw

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<<Main>$>g__TestExecutor7|0_6(<>c__DisplayClass0_0&)

About this issue

Original URL
State: closed
Created a year ago
Comments: 36 (36 by maintainers)

Most upvoted comments

@markples - I believe this can be closed now. My original mitigation just reduced the number of exceptions internally thrown and caught during Crossgen2 compilation and so reduced the repro rate of this non-deterministic race condition. As we’re now at Preview 3 containing the proper fix for the race condition according to JanK’s explanation, we should be good now.

trylek on Apr 17, 2023

@JulieLeeMSFT - my change made Crossgen2 throw much fewer exceptions so we no longer see the occasional exception handling failures on arm64. According to my understanding of @jkotas’ explanation the underlying problem is a race condition in exception handling that should be fixed by rolling forward to SDK preview 2 as the LKG version used by the runtime repo. In my runtime repo clone from earlier today,

dotnet --version

still yields

8.0.100-preview.1.23115.2

so I suspect more work may be needed to fully fix this.

trylek on Apr 4, 2023

@jkotas commented on this yesterday in the issue thread

https://github.com/dotnet/runtime/issues/77820

Apparently the problem is understood and caused by a race condition that has been fixed since but we need to roll forward to SDK Preview 2 as the LKG version used for executing Crossgen2 to get this fixed completely. As the cause of the crash is now understood, I’ll put up a PR fixing the primary cause of the exception - missing reference to xunit assemblies when running Crossgen2 to build the tests - and that should mitigate the problem. (I didn’t want to fix this before the crash is understood as otherwise the fixed references would basically hide this error.)

trylek on Mar 14, 2023

@JulieLeeMSFT - For the problem regarding “Send to Helix” not reporting the merged JIT/Methodical work item as failed, I think this is likely a bug in the generated merged test wrapper, it probably returns 0 while it should return an exit code if any of the component tests have failed, adding @jkoritzinsky to confirm.

For the actual bug in the sizeof64_Target_64Bit_and_arm.cmd, I’m trying to repro that locally. At the first glance it looks like an OOM or invalid allocation size. As you can easily imagine, running the component tests in-proc means that in exceptional cases the failure may only be reproducible when running the entire merged test set, not just a single test, as it may be caused by some interaction between the individual test (e.g. excessive GC allocation or not shutting down a worker thread in an earlier test).

trylek on Jan 26, 2023