runtime: [tests] System.Text.Json.Tests segfault, for `Libraries Test Run release coreclr OSX x64 Release`

I’ve been able to catch 3 crash dumps so far, all of them have the bad object right in front of a pinned plug. In two cases, the size of the bad object was 32 bytes, in one case it was 24 bytes (a boxed System.Boolean). I wonder if we can simply generalize your fix for is_plug_padded, clearing the bits when we save to saved_pre_plug or saved_post_plug, but not for the copy saved to saved_pre_plug_reloc or saved_post_plug_reloc.

cc @stephentoub in case he’s interested in how easy it is to get to dumps on all OS’es by just copy/pasting a couple commands out the log.

We have some fresh dumps for this now, and thanks to recent effort by @safern, getting set up debugging them is practically a single command.

I used this query –

https://engsrvprod.kusto.windows.net/engineeringdata?query=H4sIAAAAAAAEAJWSXWvCMBSG7wX%2FQ8hVC127zSGD0YH7EPRibLixyxGb4zza5pTkFBX241erpa3zZncpeZPzvE%2BTAouFwhS0iMWU5q7fi6IfsVmCBTGjwiYgEjKs0DghNbEBjmxhGDOQ%2Fd5JMo6FTDDKi3mKSdRNRxYWLlqC0i7KlGOwUpD9%2F0k0cs8o6tGvlnKwjOBaoMkSkjXoFiFTShuw3lsBBYxSVM5v8vWmJLeVfqsWK8ulmnuhvskbXOpqL7e0goT3uiY6EKK5MqjrlHxBRebiXFkHXytHxmtQ%2FaDtuQqGT7BAg4xkJrryMbwdVgVWhEas0egYjQHb73mfZNcThsw1qGOLYHS6e1HZweZsVzrOwnfYcjgtp5crx67rriKvTvxVMUpzNBAOrq7DUaaHNydeuHDVmAeln7fI8oyYmnK%2FPpoMxCMZRyl8WPQFmUP0rt8bl0%2Bw3ab87HLJJW0umFp%2FtL6%2B07zJn%2FY%2Fr%2FL49kuShvYXf5f9RRQDAAA%3D&web=0

let failed = Jobs //| where Source contains “dotnet/runtime” | where Source == “ci/public/dotnet/runtime/refs/heads/master” or Source == “ci/public/dotnet/runtime/refs/heads/main” // | where Properties contains “checked” | where tolower(QueueAlias) contains tolower(“osx”) | where Started > ago(30d) | project JobId, QueueAlias, Source // , Props=parse_json(Properties), //| where Props.DefinitionId == “686” | join kind=inner (WorkItems | where FriendlyName == “System.Text.Json.Tests” // | where QueueName contains tolower(“Alpine.312.Amd64”) | where Status == “BadExit” | project JobId, WorkItemId, Started, ConsoleUri) on JobId; Files | where FileName contains “how-to” | where WorkItemFriendlyName contains “System.Text.Json” | join kind=inner failed on WorkItemId

And I clicked the link in the most recent hit, it pulls up how-to-debug-dump.md, which gives me the magic commands to paste in to get dump, symbols, and binaries pulled down, then open it in lldb, eg

dotnet tool install --global runfo
dotnet tool update --global runfo
runfo get-helix-payload -j a18c2113-8e70-40f3-967c-ada5df3ab122 -w System.Text.Json.Tests -o ~/helix_payload/System.Text.Json.Tests\
lldb --core ~/helix_payload/System.Text.Json.Tests/workitems/System.Text.Json.Tests/core.41566 ~/helix_payload/System.Text.Json.Tests/shared/Microsoft.NETCore.App/6.0.0/dotnet

It really is super easy now. Unfortunately I don’t have a Mac, so I couldn’t actually do the lldb part. @steveharter could you please take a look, and tell us how this goes for you?

cc @jkotas