runtime: System.Text.Json.SourceGeneration intermittent failures

Frequency:

  • 4/1-7/22 … 618 failures, first one starting on 7/5 … 36 failures per day. Many of them in Official runs
let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests('System.Text.Json.SourceGeneration', '', true, 'NaN', true);

Assertions failed in several builds for tests:

Example failure: Assert.Equal() Failure Expected: 1.10000002 Actual: NaN log results

cc @jtschuster

Runfo Tracking Issue: System.Text.Json failures on Mono_Release

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900783 runtime-extra-platforms PR 67861 net7.0-tvOS-Release-arm64-Mono_Release-OSX.1100.Amd64.AppleTV.Open console.log runclient.py
1900783 runtime-extra-platforms PR 67861 net7.0-iOSSimulator-Release-x64-Mono_Release-OSX.1200.Amd64.Open console.log runclient.py
1900774 runtime-extra-platforms PR 72545 net7.0-tvOS-Release-arm64-Mono_Release-OSX.1100.Amd64.AppleTV.Open console.log runclient.py
1900774 runtime-extra-platforms PR 72545 net7.0-tvOSSimulator-Release-x64-Mono_Release-OSX.1200.Amd64.Open console.log runclient.py
1900571 runtime-extra-platforms Rolling net7.0-tvOS-Release-arm64-Mono_Release-OSX.1100.Amd64.AppleTV.Open console.log runclient.py
1900571 runtime-extra-platforms Rolling net7.0-tvOSSimulator-Release-x64-Mono_Release-OSX.1200.Amd64.Open console.log runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900134 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900062 runtime-extra-platforms Rolling net7.0-tvOS-Release-arm64-Mono_Release-OSX.1015.Amd64.AppleTV.Open console.log test results runclient.py
1900062 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900062 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900062 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899117 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899117 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898170 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898170 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898170 runtime-extra-platforms Rolling net7.0-MacCatalyst-Release-arm64-Mono_Release-OSX.1100.Arm64.Open console.log runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py

Displaying 100 of 1291 results

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
5 59 97

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 21 (21 by maintainers)

Most upvoted comments

@danmoseley we do want to test json on wasm but the threaded runtime lanes are just being brought up so we expect breakage there.

@simonrozsival can you work with @radical to disable the tests for threaded lanes?

+2
lewing on Jul 21, 2022

@radical I’ll look into it on Monday

+1
simonrozsival on Jul 22, 2022

@karelz https://github.com/dotnet/runtime/pull/72664

+1
lewing on Jul 22, 2022

It’d be great if someone could add a skip for x86 Android as you suggest.

+1
danmoseley on Jul 21, 2022
Read more comments on GitHub
← runtime: System.Text.Json's generator causes considerable package size increase
runtime: System.Text.Json.SourceGeneration.UnitTests.GeneratorTests failing with OutOfMemoryException on net7.0-windows-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open →