runtime: [release/7.0][wasm] Multiple `JIT.*` runtime test failures on rolling builds

Occurrences:

Day	Run	OS	Details
9/23	Rolling run 28406 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	1x JIT.* failures
9/23	Rolling run 28106 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	3x JIT.* failures
9/23	Rolling run 27711 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	3x JIT.* failures
9/22	Rolling run 27252 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	3x JIT.* failures
9/22	Rolling run 25951 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	4x JIT.* failures
9/21	Rolling run 25763 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	2x JIT.* failures
9/21	Rolling run 25169 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	8x JIT.* failures
9/20	Rolling run 23284 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	2x JIT.* failures
9/19	Rolling run 22992 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	5x JIT.* failures
9/19	Rolling run 22495 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	2x JIT failures & 1x Loader failure
9/17-9/18	no runs
9/16	Rolling run 20657 (7.0-rc2)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	2x JIT.* failures
9/16	Rolling run 20304 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	3x JIT.* failures
9/16	Rolling run 20115 (7.0-rc2)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	1x Loader.* failure
9/15	Rolling run 18463 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	1x JIT.* failure
9/15	Rolling run 18930 (7.0-rc2)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	2x JIT.* failures

Original report

Failing build.

JIT.HardwareIntrinsics.Arm.AdvSimd:

    JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.sh [FAIL]
      
      Return code:      1
      Raw output file:      /datadisks/disk1/work/B2CE09D6/w/ABBA093D/uploads/Reports/JIT.HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.output.txt
      Raw output:
      BEGIN EXECUTION
      MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
        AppDir: /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/WasmApp/
        TestBinDir: /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r
        ArtifactsBinDir: 
        Generated app bundle at /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/WasmApp/
      /datadisks/disk1/work/B2CE09D6/p/dotnet-cli/sdk/7.0.100-rc.1.22425.9/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(219,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj]
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/B2CE09D6/p
      > /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_HardwareIntrinsics._Arm_AdvSimd_AdvSimd_Part6_r_AdvSimd_Part6_r_._Arm_AdvSimd_AdvSimd_Part6_r_AdvSimd_Part6_r_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      1
        Raw output file:      /datadisks/disk1/work/B2CE09D6/w/ABBA093D/uploads/Reports/JIT.HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.output.txt
        Raw output:
        BEGIN EXECUTION
        MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
          AppDir: /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/WasmApp/
          TestBinDir: /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r
          ArtifactsBinDir: 
          Generated app bundle at /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/WasmApp/
        /datadisks/disk1/work/B2CE09D6/p/dotnet-cli/sdk/7.0.100-rc.1.22425.9/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(219,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj]
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/B2CE09D6/p
        > /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.sh

JIT.HardwareIntrinsics.X86.General:

    JIT/HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.sh [FAIL]
      
      Return code:      1
      Raw output file:      /datadisks/disk1/work/B2CE09D6/w/AF3E0916/uploads/Reports/JIT.HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.output.txt
      Raw output:
      BEGIN EXECUTION
      MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
      /datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj : error : Failed to resolve SDK 'Microsoft.Build.NoTargets/1.0.53'. Package restore was successful but a package with the ID of "Microsoft.Build.NoTargets" was not installed.
      /datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj : error MSB4236: The SDK 'Microsoft.Build.NoTargets/1.0.53' specified could not be found.
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/B2CE09D6/p
      > /datadisks/disk1/work/B2CE09D6/w/AF3E0916/e/JIT/HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_HardwareIntrinsics._X86_General_VectorUnused_r_VectorUnused_r_._X86_General_VectorUnused_r_VectorUnused_r_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      1
        Raw output file:      /datadisks/disk1/work/B2CE09D6/w/AF3E0916/uploads/Reports/JIT.HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.output.txt
        Raw output:
        BEGIN EXECUTION
        MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
        /datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj : error : Failed to resolve SDK 'Microsoft.Build.NoTargets/1.0.53'. Package restore was successful but a package with the ID of "Microsoft.Build.NoTargets" was not installed.
        /datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj : error MSB4236: The SDK 'Microsoft.Build.NoTargets/1.0.53' specified could not be found.
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/B2CE09D6/p
        > /datadisks/disk1/work/B2CE09D6/w/AF3E0916/e/JIT/HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.sh
  Finished:    JIT.HardwareIntrinsics.XUnitWrapper

JIT.HardwareIntrinsics.X86.Ssse3:

      Unhandled exception: System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-aCL7KKV98jx3O3fvRayafjy8_eCkKjpwtyjOVymKcEE'
         at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
         at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
         at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
         at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)
      
      Return code:      82
      Raw output file:      /datadisks/disk1/work/B2CE09D6/w/B25B099A/uploads/Reports/JIT.HardwareIntrinsics/X86/Ssse3/Ssse3_ro/Ssse3_ro.output.txt
...

JIT.Regression.CLR-x86-JIT.V1-M12-M13:

    JIT/Regression/CLR-x86-JIT/V1-M13-RTM/b92289/b92289/b92289.sh [FAIL]
      Unhandled exception: System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-aCL7KKV98jx3O3fvRayafjy8_eCkKjpwtyjOVymKcEE'
         at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
         at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
         at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
         at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)

... followed by lot of failures ...

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 38 (38 by maintainers)

Commits related to this issue

[wasm] Don't use MS.Build.NoTargets SDK for runtime tests This will also avoid intermittent failures like https://github.com/dotnet/runtime/issues/75391 — committed to radical/runtime by radical 2 years ago
[wasm] Don't use MS.Build.NoTargets SDK for runtime tests (#75939) This will also avoid intermittent failures like https://github.com/dotnet/runtime/issues/75391 — committed to dotnet/runtime by radical 2 years ago
[wasm] Don't use MS.Build.NoTargets SDK for runtime tests This will also avoid intermittent failures like https://github.com/dotnet/runtime/issues/75391 — committed to dotnet/runtime by radical 2 years ago
[wasm] Don't use MS.Build.NoTargets SDK for runtime tests (#76107) This will also avoid intermittent failures like https://github.com/dotnet/runtime/issues/75391 Co-authored-by: Ankit Jain <radica... — committed to dotnet/runtime by github-actions[bot] 2 years ago
global.json: Bump sdk to 7.0.100-rtm.22476.1 to pick up a msbuild fix Context: https://github.com/dotnet/runtime/issues/75391#issuecomment-1258304105 — committed to radical/runtime by radical 2 years ago
[wasm] Don't use MS.Build.NoTargets SDK for runtime tests (#75939) This will also avoid intermittent failures like https://github.com/dotnet/runtime/issues/75391 # Conflicts: # src/tests/Common/wasm-... — committed to dotnet/runtime by radical 2 years ago
[release/6.0] Update dependencies from dotnet/emsdk dotnet/arcade (#78062) * Update dependencies from https://github.com/dotnet/emsdk build 20221108.3 Microsoft.NET.Workload.Emscripten.Manifest-6.... — committed to dotnet/runtime by dotnet-maestro[bot] 2 years ago

Most upvoted comments

I’m backporting https://github.com/dotnet/runtime/pull/76107 which would remove the whole NoTargets issue.

radical on Sep 23, 2022

There have been multiple roslyn/msbuild issues (and some caused by ubuntu’s openSSL). I say we take https://github.com/dotnet/runtime/pull/76090 for now and see if things go green. The failures here are definitely build related, if op_Equality was actually failing intermittently we’d see it in all the tests not just these.

lewing on Sep 23, 2022

@carlossanlop we will need to figure out better way to track occurrences. It is super-hard to figure out which builds are reported, how often it happened, etc.

That’s what I said originally 😦 too many issues grouped into a single one. If I have time, I’ll split them into separate ones later.

carlossanlop on Sep 23, 2022