runtime: System.Text.Json failing some large file tests
If I didn’t know any better, I’d say “memory corruption”.
Console log: 'System.Text.Json.Tests' from job ff748885-0016-49f2-ad26-20905cd95f3d workitem 0045d34b-f5a3-424a-8789-b58e57023e81 (osx.1015.amd64.open) executed on machine dci-mac-build-195
+ ./RunTests.sh --runtime-path /tmp/helix/working/A15608F4/p
----- start Mon Sep 27 11:49:52 EDT 2021 =============== To repro directly: =====================================================
pushd .
/tmp/helix/working/A15608F4/p/dotnet exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
popd
===========================================================================================================
/private/tmp/helix/working/A15608F4/w/A1BD08B8/e /private/tmp/helix/working/A15608F4/w/A1BD08B8/e
Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
Discovered: System.Text.Json.Tests (found 2776 of 2815 test cases)
Starting: System.Text.Json.Tests (parallel test collections = on, max threads = 12)
System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes(formatted: False, skipValidation: False) [FAIL]
Assert.Equal() Failure
↓ (pos 1102164)
Expected: ···qKioqKioqKioqKioqKiosKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq···
Actual: ···qKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq···
↑ (pos 1102164)
Stack Trace:
/_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(738,0): at System.Text.Json.JsonTestHelper.AssertContentsAgainstJsonNet(String expectedValue, String value, Boolean skipSpecialRules)
/_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(714,0): at System.Text.Json.JsonTestHelper.AssertContents(String expectedValue, ArrayBufferWriter`1 buffer, Boolean skipSpecialRules)
/_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Utf8JsonWriterTests.cs(3202,0): at System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes(Boolean formatted, Boolean skipValidation)
System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(formatted: False, skipValidation: False, keyString: ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>") [FAIL]
Assert.Equal() Failure
↓ (pos 484474)
Expected: ···e\u003e\u003e\u003e\w003e\u003e\u003e\u003e\u003e\u003e\u003e···
Actual: ···e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e···
↑ (pos 484474)
Stack Trace:
/_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(738,0): at System.Text.Json.JsonTestHelper.AssertContentsAgainstJsonNet(String expectedValue, String value, Boolean skipSpecialRules)
/_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/JsonTestHelper.cs(714,0): at System.Text.Json.JsonTestHelper.AssertContents(String expectedValue, ArrayBufferWriter`1 buffer, Boolean skipSpecialRules)
/_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Utf8JsonWriterTests.cs(5617,0): at System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(Boolean formatted, Boolean skipValidation, String keyString)
System.Text.Json.Serialization.Tests.StreamTests_Sync.HandleCollectionsAsync [FAIL]
Assert.Equal() Failure
↓ (pos 12025)
Expected: ···eeeeeeeeeeeeeeeeeeeegeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee···
Actual: ···eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee···
↑ (pos 12025)
Stack Trace:
/_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(61,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
/_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
/_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(45,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(JsonElement expected, JsonElement actual)
/_/src/libraries/System.Text.Json/tests/Common/JsonTestHelper.cs(20,0): at System.Text.Json.JsonTestHelper.AssertJsonEqual(String expected, String actual)
/_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(102,0): at System.Text.Json.Serialization.Tests.StreamTests.TestDeserialization[TElement](Stream memoryStream, String expectedJson, Type type, JsonSerializerOptions options)
/_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(71,0): at System.Text.Json.Serialization.Tests.StreamTests.PerformSerialization[TElement](Object obj, Type type, JsonSerializerOptions options)
/_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(53,0): at System.Text.Json.Serialization.Tests.StreamTests.RunTestAsync[TElement]()
/_/src/libraries/System.Text.Json/tests/System.Text.Json.Tests/Serialization/Stream.Collections.cs(25,0): at System.Text.Json.Serialization.Tests.StreamTests.HandleCollectionsAsync()
--- End of stack trace from previous location ---
Finished: System.Text.Json.Tests
=== TEST EXECUTION SUMMARY ===
System.Text.Json.Tests Total: 20247, Errors: 0, Failed: 3, Skipped: 0, Time: 58.695s
- System.Text.Json.Serialization.Tests.StreamTests_Sync.HandleCollectionsAsync
- System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes(formatted: False, skipValidation: False)
- System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(formatted: False, skipValidation: False, keyString: “>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>”)
Error Message
Fill the error message using known issues guidance.
{
"ErrorMessage": "System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers",
"BuildRetry": false
}
Report
| Build | Definition | Test | Pull Request |
|---|---|---|---|
| 508335 | dotnet/runtime | WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution | dotnet/runtime#95292 |
| 505651 | dotnet/runtime | System.Text.Json.Tests.WorkItemExecution | |
| 501644 | dotnet/runtime | normal-System.Text.Json.Tests.WorkItemExecution | dotnet/runtime#95292 |
| 498760 | dotnet/runtime | normal-System.Text.Json.Tests.WorkItemExecution | dotnet/runtime#95985 |
| 497560 | dotnet/runtime | System.Text.Json.Tests.WorkItemExecution | |
| 497130 | dotnet/runtime | System.Text.Json.Tests.WorkItemExecution | |
| 495924 | dotnet/runtime | System.Text.Json.Tests.WorkItemExecution | |
| 491743 | dotnet/runtime | System.Text.Json.Tests.WorkItemExecution | dotnet/runtime#95733 |
| 489669 | dotnet/runtime | System.Text.Json.Tests.WorkItemExecution | |
| 485217 | dotnet/runtime | normal-System.Text.Json.Tests.WorkItemExecution | dotnet/runtime#95469 |
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 0 | 2 | 10 |
Known issue validation
Build: 🔎 Result validation: ⚠️ Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the “Build: 🔎” line.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 25 (25 by maintainers)
Ok, finally got lucky - random Assert.Contains failed where
actualString.IndexOf(expectedSubstring, StringComparison.CurrentCulture)returned -1 butactualString.IndexOf(expectedSubstring, StringComparison.Ordinal)returned non-negative value.IMO some issues we’re seeing are some kind of globalization issue - maybe cultures didn’t get initialized correctly when it happens on multiple threads. We’ll likely need to try to write some standalone repro and try to attach debugger to see if we can see what exactly happened and then maybe we can guess what should be changed.
From JSON perspective we likely should change to ordinal compare anyway but it would be good to first nail down what the problem with CurrentCulture is
@krwq perhaps you can put it in a shared place - AssertExtensions, we already did something very similar for Equals
https://github.com/dotnet/runtime/pull/67586/files#diff-9f9dc769fe4cfd311f1e57c7572eaeeb09c0c72ffe633131ed0fd8b3ed024994R505
If we build up several of these improved assertions and they show value, we could try to see whether xunit would be interested in any changes.
I’ll add couple of other tests I’ve seen locally, first 3 look like the “not found” value is actually in the string and in all 3 cases it’s using current culture for comparison:
also in another run I’ve seen those 2 happen in a single run which was rather unusual and I think the stream was actually corrupted (somehow ended up with first byte being zero) - interestingly this only affected async related cases and same test using non-async code path was fine:
I’ll be replacing Assert.Contains in our tests into something else which can possibly provide us some more details