runtime: Multiple JIT Stress test failures

While working on https://github.com/dotnet/coreclr/pull/16957 I found there were multiple unrelated tests that were failing.

General Failure

Jobs with many failures

baseservices_threading._mutex_misc_mutexctor1_mutexctor1

https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_gcstress0xc_minopts_heapverify1_prtest/10/testReport/baseservices_threading/_mutex_misc_mutexctor1_mutexctor1_/_mutex_misc_mutexctor1_mutexctor1_cmd/

	Assert failure(PID 9080 [0x00002378], Thread: 2304 [0x0900]): CONTRACT VIOLATION by Frame::Push at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\frames.cpp\" @ 394

	MODE_COOPERATIVE encountered while thread is in preemptive state.

	 CONTRACT in Frame::Push at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\frames.cpp\" @ 394
	 OVERRIDE_TYPE_LOAD_LEVEL_LIMIT in MethodDescCallSite::CallTargetWorker at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\callhelpers.cpp\" @ 371
	 CONTRACT in MethodDescCallSite::CallTargetWorker at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\callhelpers.cpp\" @ 361
	 CONTRACT in ThreadNative::KickOffThread_Worker at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\comsynchronizable.cpp\" @ 206
	 CONTRACT in ManagedThreadBase_DispatchInner at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\threads.cpp\" @ 8816
	 CONTRACT in ManagedThreadBase_FullTransitionWithAD at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\threads.cpp\" @ 9197
	 CONTRACT in ThreadNative::KickOffThread at \"d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\comsynchronizable.cpp\" @ 305

	We can't find the violated contract. Look for an old-style non-holder-based contract.

	CORECLR! CONTRACT_ASSERT + 0x458 (0x00007ffe`f1db47f8)
	CORECLR! EEContract::DoChecks + 0x252 (0x00007ffe`f1ed1032)
	CORECLR! Frame::Push + 0xD4 (0x00007ffe`f216ecd4)
	CORECLR! LinkFrameAndThrow + 0x32 (0x00007ffe`f1ed0d92)
	CORECLR! NakedThrowHelper2 + 0x9 (0x00007ffe`f23981b9)
	CORECLR! NakedThrowHelper_RspAligned + 0x1E (0x00007ffe`f23981eb)
	CORECLR! NakedThrowHelper_FixRsp + 0x5 (0x00007ffe`f23981f5)
	SYSTEM.PRIVATE.CORELIB! <no symbol> + 0x0 (0x00007ffe`f1811e90)
	SYSTEM.PRIVATE.CORELIB! <no symbol> + 0x0 (0x00007ffe`f18f0ed4)
	<no module>! <no symbol> + 0x0 (0x00007ffe`92ce616f)
	 File: d:\\j\\workspace\\x64_checked_w---e8fbcb57\\src\\vm\\frames.cpp Line: 394
	 Image: D:\\j\\workspace\\x64_checked_w---e8fbcb57\\bin\\tests\\Windows_NT.x64.Checked\\Tests\\Core_Root\\CoreRun.exe

JIT_Methodical.refany__il_relarray3__il_relarray3

https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x86_checked_windows_nt_gcstress0xc_minopts_heapverify1_prtest/17/testReport/junit/JIT_Methodical/_refany__il_relarray3__il_relarray3_/_refany__il_relarray3__il_relarray3_cmd/

	Assert failure(PID 5580 [0x000015cc], Thread: 7364 [0x1cc4]): !hasPartialArgInfo || isZero(argHigh) || (argHigh == CONSTRUCT_ptrArgTP(1, (argCnt-1)))

	CORECLR! scanArgRegTableI + 0x5F3 (0x7261e063)
	CORECLR! EECodeManager::EnumGcRefs + 0x322 (0x72617a62)
	CORECLR! GcStackCrawlCallBack + 0x169 (0x727eda29)
	CORECLR! Thread::MakeStackwalkerCallback + 0x45 (0x7246b345)
	CORECLR! Thread::StackWalkFramesEx + 0x177 (0x7246c9dc)
	CORECLR! Thread::StackWalkFrames + 0x15E (0x7246c7e2)
	CORECLR! ScanStackRoots + 0x18C (0x728f74da)
	CORECLR! GCToEEInterface::GcScanRoots + 0x10F (0x728f6adc)
	CORECLR! WKS::gc_heap::mark_phase + 0x1B1 (0x728c1c2e)
	CORECLR! WKS::gc_heap::gc1 + 0x133 (0x728bc800)
	 File: d:\\j\\workspace\\x86_checked_w---9da75386\\src\\vm\\eetwain.cpp Line: 2658
	 Image: D:\\j\\workspace\\x86_checked_w---9da75386\\bin\\tests\\Windows_NT.x86.Checked\\Tests\\Core_Root\\CoreRun.exe

JIT_Performance.CodeQuality_BenchmarksGame_regex_redux_regex_redux_5_regex_redux_5

https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x86_checked_windows_nt_gcstress0xc_zapdisable_heapverify1_prtest/8/testReport/JIT_Performance/_CodeQuality_BenchmarksGame_regex_redux_regex_redux_5_regex_redux_5_/_CodeQuality_BenchmarksGame_regex_redux_regex_redux_5_regex_redux_5_cmd/

	Assert failure(PID 6868 [0x00001ad4], Thread: 760 [0x02f8]): Consistency check failed: hit privileged instruction!FAILED: !ExecutionManager::IsManagedCode(GetIP(pContext))
	 File: d:\\j\\workspace\\x86_checked_w---a898fbbe\\src\\vm\\excep.cpp Line: 6944
	 Image: D:\\j\\workspace\\x86_checked_w---a898fbbe\\bin\\tests\\Windows_NT.x86.Checked\\Tests\\Core_Root\\CoreRun.exe

JIT_HardwareIntrinsics.X86_Sse2_ConvertToDouble_r_ConvertToDouble_r

https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_jitstressregs4_prtest/31/testReport/junit/JIT_HardwareIntrinsics/_X86_Sse2_ConvertToDouble_r_ConvertToDouble_r_/_X86_Sse2_ConvertToDouble_r_ConvertToDouble_r_cmd/

	Assert failure(PID 16440 [0x00004038], Thread: 11024 [0x2b10]): Assertion failed '(unsigned)ins < _countof(insCodesRM)' in 'IntelHardwareIntrinsicTest.Program:Main(ref):int' (IL size 243)

	 File: d:\\j\\workspace\\x64_checked_w---db056dfb\\src\\jit\\emitxarch.cpp Line: 1249
	 Image: D:\\j\\workspace\\x64_checked_w---db056dfb\\bin\\tests\\Windows_NT.x64.Checked\\Tests\\Core_Root\\CoreRun.exe

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 16 (16 by maintainers)

Commits related to this issue

Most upvoted comments

Similar looking failure in regex-redux-5:

(276c.4d3c): Unknown exception - code c0000096 (first chance)
(276c.4e88): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
System_Private_CoreLib!System.Reflection.Emit.DynamicILGenerator.GetTokenFor(System.Reflection.RuntimeFieldInfo, System.RuntimeType)+0x1b:
00007ff9`58d275cb ff5030          call    qword ptr [rax+30h] ds:00007ff9`5889ab20={System.Reflection.RtFieldInfo.get_FieldHandle() (00007ff9`58777618)}

Both these tests are heavily multi-threaded so would guess we are hitting some kind of race condition in gcstress here. @swaroop-sridhar does this look at all familiar?

Suggest we disable these from running in stress for now.

+1
AndyAyersMS on Mar 22, 2018

@4creators, that one has been failing since https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_jitstressregs4/465/

It may have not been running before since the JIT Stress Reg jobs don’t set EnableIncompleteISAClass.

@CarolEidt, @fiigii, we should ensure we run the full suite of JIT Stress jobs (Reg, GC, etc) on any PR which marks an ISA as complete.

+1
tannergooding on Mar 19, 2018
Read more comments on GitHub
← runtime: Multicast doesn't work with XCode 12.5 and higher on iOS
runtime: Multiple 'System.OutOfMemoryException' errors in .NET 7 →