runtime: [libraries-pgo] System.Tests.AppDomainTests.AssemblyLoad fails under jitosr_stress_random

Error Blob

{
  "ErrorMessage": "System.Tests.AppDomainTests.AssemblyLoad [FAIL]",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

See e.g. https://dev.azure.com/dnceng-public/public/_build/results?buildId=447127&view=ms.vss-test-web.build-test-results-tab&runId=10007342&resultId=152206&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab

DOTNET_JitRandomOnStackReplacement=15
DOTNET_OSR_HitLimit=2
DOTNET_TC_OnStackReplacement=1
DOTNET_TC_OnStackReplacement_InitialCounter=1
DOTNET_TC_QuickJitForLoops=1
DOTNET_TieredCompilation=1
...
C:\h\w\A316094E\w\A8060938\e>"C:\h\w\A316094E\p\dotnet.exe" exec --runtimeconfig System.Runtime.Extensions.Tests.runtimeconfig.json --depsfile System.Runtime.Extensions.Tests.deps.json xunit.console.dll System.Runtime.Extensions.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Runtime.Extensions.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Extensions.Tests (found 981 of 1040 test cases)
  Starting:    System.Runtime.Extensions.Tests (parallel test collections = on, max threads = 4)

Assert failure(PID 2968 [0x00000b98], Thread: 2540 [0x09ec]): Consistency check failed: AV in clr at this callstack:
------
CORECLR! GetCLRRuntimeHost + 0x11E130 (0x00007ffa`248a79a0)
CORECLR! <no symbol> + 0x0 (0x00007ffa`24777779)
CORECLR! GetCLRRuntimeHost + 0x3BB926 (0x00007ffa`24b45196)
CORECLR! GetCLRRuntimeHost + 0x3B9595 (0x00007ffa`24b42e05)
CORECLR! GetCLRRuntimeHost + 0x3975AC (0x00007ffa`24b20e1c)
<no module>! <no symbol> + 0x0 (0x00007ff9`c57930ed)
<no module>! <no symbol> + 0x0 (0x00000070`fb974230)
<no module>! <no symbol> + 0x0 (0x00000070`fb973e98)
<no module>! <no symbol> + 0x0 (0x00000070`fb974538)
CORECLR! coreclr_shutdown_2 + 0x2ACDB8 (0x00007ffa`2539d608)
<no module>! <no symbol> + 0x0 (0x00000070`fb973910)
<no module>! <no symbol> + 0x0 (0x00000174`59f18de0)
<no module>! <no symbol> + 0x0 (0x00000000`000016ad)
<no module>! <no symbol> + 0x0 (0x00000070`fb9739d0)
<no module>! <no symbol> + 0x0 (0x00000070`fb973840)
-----
.AV on tid=0x9ec (2540), cxr=00000070FB9723C0, exr=00000070FB9728B0
FAILED: false

CORECLR! GetCLRRuntimeHost + 0x7BB0E6 (0x00007ffa`24f44956)
CORECLR! GetCLRRuntimeHost + 0x402A5 (0x00007ffa`247c9b15)
CORECLR! GetCLRRuntimeHost + 0x3FB81 (0x00007ffa`247c93f1)
CORECLR! GetCLRRuntimeHost + 0x3FAE9 (0x00007ffa`247c9359)
CORECLR! GetCLRRuntimeHost + 0x40525 (0x00007ffa`247c9d95)
NTDLL! RtlInitializeCriticalSection + 0x190 (0x00007ffa`556b6b40)
NTDLL! RtlImageNtHeaderEx + 0x17B (0x00007ffa`556846cb)
NTDLL! KiUserExceptionDispatcher + 0x3A (0x00007ffa`556f989a)
CORECLR! GetCLRRuntimeHost + 0x11E130 (0x00007ffa`248a79a0)
CORECLR! <no symbol> + 0x0 (0x00007ffa`24777779)
    File: D:\a\_work\1\s\src\coreclr\vm\excep.cpp Line: 7177
    Image: C:\h\w\A316094E\p\dotnet.exe

    System.Tests.AppDomainTests.AssemblyLoad [FAIL]
      Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Exit code was -1073740286 but it should have been 42
      Stack Trace:

Fails on multiple platforms, including win-x64

cc @AndyAyersMS

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=447127 Error message validated: System.Tests.AppDomainTests.AssemblyLoad [FAIL] Result validation: ✅ Known issue matched with the provided build. Validation performed at: 10/23/2023 8:16:38 AM UTC

Report

Build	Definition	Test	Pull Request
460357	dotnet/runtime	System.Tests.AppDomainTests.AssemblyLoad
460101	dotnet/runtime	System.Tests.AppDomainTests.AssemblyLoad
454731	dotnet/runtime	System.Tests.AppDomainTests.AssemblyLoad
454338	dotnet/runtime	System.Tests.AppDomainTests.AssemblyLoad
447127	dotnet/runtime	System.Tests.AppDomainTests.AssemblyLoad

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	2	5