runtime: mono `minijit` runtime tests failing with `coreclr_initialize failed - Error: 0x80004005`

Failures 4/5-9/8 (incl. PRs and 7.0 branches):

Day Run OS
9/5 Rolling run 5191 (only in Runfo) mono Linux arm64 Release @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open
9/5 Rolling run 4698 (only in Runfo) mono Linux arm64 Release @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open
8/30 Rolling run Ubuntu.1804.Arm64.Open - Mono Interpreter
8/30 PR #74831 OSX.1200.Amd64.Open - Mono Interpreter
8/24 PR #74459 OSX 12 amd64 - Moni minijit
8/8 PR #73416 OSX 12 amd64 - Mono Interpreter
8/4 PR #73230 OSX 12 amd64 - Mono Interpreter
8/2 PR #73157 OSX 12 amd64 - Moni minijit
7/29 Rolling run Ubuntu.1804 arm64 - Mono llvm full aot
7/12-7/27 8x unique PRs OSX 12 amd64 and Ubuntu.1804 arm64
6/1-6/30 6x unique PRs OSX 12 amd64 and Ubuntu.1804 arm64
5/1-5/31 2x unique PRs OSX 12 amd64
4/1-4/30 6x unique PRs OSX 12 amd64 and Ubuntu.1804 arm64
let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (Branch startswith 'refs/heads/release/7.0') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests('JIT/Math/Functions/Functions_ro', '', true, 'coreclr_initialize', true);

Hit on https://github.com/dotnet/runtime/pull/72094 . Build. From one of the logs:

    JIT/Math/Functions/Functions_ro/Functions_ro.sh [FAIL]
      BEGIN: coreclr_initialize failed - Error: 0x80004005
      END: coreclr_initialize failed - Error: 0x80004005
      
      Return code:      1
      Raw output file:      /tmp/helix/working/B5FB0986/w/AC37094B/uploads/Reports/JIT.Math/Functions/Functions_ro/Functions_ro.output.txt
      Raw output:
      BEGIN EXECUTION
      /tmp/helix/working/B5FB0986/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Functions_ro.dll ''
      Exe path: /private/tmp/helix/working/B5FB0986/p/corerun
      Properties:
          TRUSTED_PLATFORM_ASSEMBLIES = <see console log for the full list>
          APP_PATHS = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/
          NATIVE_DLL_SEARCH_DIRECTORIES = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/:/tmp/helix/working/B5FB0986/p/:
          System.Reflection.Metadata.MetadataUpdater.IsSupported = false
      Managed assembly: /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.dll
      Arguments (0): 
      Expected: 100
      Actual: 255
      END EXECUTION - FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/tmp/helix/working/B5FB0986/p
      > /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_Math._Functions_Functions_ro_Functions_ro_._Functions_Functions_ro_Functions_ro_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        BEGIN: coreclr_initialize failed - Error: 0x80004005
        END: coreclr_initialize failed - Error: 0x80004005
        
        Return code:      1
        Raw output file:      /tmp/helix/working/B5FB0986/w/AC37094B/uploads/Reports/JIT.Math/Functions/Functions_ro/Functions_ro.output.txt
        Raw output:
        BEGIN EXECUTION
        /tmp/helix/working/B5FB0986/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Functions_ro.dll ''
        Exe path: /private/tmp/helix/working/B5FB0986/p/corerun
        Properties:
            TRUSTED_PLATFORM_ASSEMBLIES = <see console log for the full list>
            APP_PATHS = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/
            NATIVE_DLL_SEARCH_DIRECTORIES = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/:/tmp/helix/working/B5FB0986/p/:
            System.Reflection.Metadata.MetadataUpdater.IsSupported = false
        Managed assembly: /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.dll
        Arguments (0): 
        Expected: 100
        Actual: 255
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/tmp/helix/working/B5FB0986/p
        > /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.sh

cc @lewing

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (18 by maintainers)

Most upvoted comments

@karelz thanks for checking! We have enough information to move forward now.