runtime: Linq tests timing out in WASM leg

Linq tests appear to be timing out (or possibly hanging) in WASM

https://dev.azure.com/dnceng/public/_build/results?buildId=780635&view=ms.vss-test-web.build-test-results-tab&runId=24492990&resultId=164319&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-41038-merge-558fe2197ef6431186/System.Linq.Tests/console.01a47de4.log?sv=2019-02-02&se=2020-09-09T20%3A59%3A51Z&sr=c&sp=rl&sig=Nv2ILnn693jzHBwsZrPiM8NJrLkYK7Qe2su3kYUaY78%3D

Executed on a003RYO
+ export PATH=/home/helixbot/work/A66808D0/p/dotnet:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
+ export DOTNET_ROOT=/home/helixbot/work/A66808D0/p/dotnet
+ export DOTNET_CLI_TELEMETRY_OPTOUT=1
+ export PATH=/home/helixbot/work/A66808D0/p/xharness-cli:/home/helixbot/work/A66808D0/p/dotnet:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
+ export XHARNESS_DISABLE_COLORED_OUTPUT=true
+ export XHARNESS_LOG_WITH_TIMESTAMPS=true
+ export XHARNESS_CLI_PATH=/home/helixbot/work/A66808D0/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.20411.1/tools/netcoreapp3.1/any/Microsoft.DotNet.XHarness.CLI.dll
+ ./RunTests.sh --runtime-path /home/helixbot/work/A66808D0/p
XHarness command issued: wasm test --engine=V8 --engine-arg=--stack-trace-limit=1000 --js-file=runtime.js -v --output-directory=./xharness-output -- --run WasmTestRunner.dll System.Linq.Tests.dll -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
[21:00:14] info: test[0]
      21:00:14.6178730 v8 --expose_wasm --stack-trace-limit=1000 runtime.js -- --run WasmTestRunner.dll System.Linq.Tests.dll -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
      
[21:00:14] dbug: test[0]
      Arguments: --run,WasmTestRunner.dll,System.Linq.Tests.dll,-notrait,category=IgnoreForCI,-notrait,category=OuterLoop,-notrait,category=failing
      
[21:00:15] dbug: test[0]
      console.debug: MONO_WASM: Initializing mono runtime
      
[21:00:15] dbug: test[0]
      console.debug: MONO_WASM: ICU data archive(s) loaded, disabling invariant mode
      
[21:00:15] dbug: test[0]
      console.debug: mono_wasm_runtime_ready fe00e07a-5519-4dfe-b35a-f867dbaf2e28
      
[21:00:15] dbug: test[0]
      Initializing.....
      
[21:00:16] dbug: test[0]
      Discovering tests for System.Linq.Tests.dll
,...................

[21:15:10] dbug: test[0]
      [PASS] System.Linq.Tests.SkipLastTests.SkipLast(source: [0, 1, 2, 3, 4, ...], count: 8)
      
[21:15:12] dbug: test[0]
      [PASS] System.Linq.Tests.SkipLastTests.SkipLast(source: [0, 1, 2, 3, 4, ...], count: 13)
      
[21:15:13] dbug: test[0]
      [PASS] System.Linq.Tests.SkipLastTests.SkipLast(source: [0, 1, 2, 3, 4, ...], count: 21)
      
Killed

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 23 (23 by maintainers)

Commits related to this issue

[Wasm] Send ToArrayShouldWorkWithSpecialLengthLazyEnumerables Linq test to the outer loop Addresses the occasional timeouts in System.Linq tests. The last 3-4 iterations of ToArrayShouldWorkWithSpec... — committed to steveisok/runtime by deleted user 4 years ago
[Wasm] Send ToArrayShouldWorkWithSpecialLengthLazyEnumerables Linq test to the outer loop Addresses the occasional timeouts in System.Linq tests. The last 3-4 iterations of ToArrayShouldWorkWithSpec... — committed to dotnet/runtime by deleted user 4 years ago
[Wasm] Reduce max length on browser for ToArrayShouldWorkWithSpecialLengthLazyEnumerables Linq test (#41241) Addresses the occasional timeouts in System.Linq tests. The last 3-4 iterations of ToArra... — committed to dotnet/runtime by steveisok 4 years ago
[release/5.0] [Wasm] Reduce max length on browser for ToArrayShouldWorkWithSpecialLengthLazyEnumerables Linq test (#41268) Addresses the occasional timeouts in System.Linq tests. The last 3-4 iterat... — committed to dotnet/runtime by github-actions[bot] 4 years ago
Runtime ingestion (#73) * Applied BinaryPrimitives throughout CoreLib (#37582) * Flaky multiple HTTP/2 connection tests read full request body (#40844) There are 2 flaky HTTP/2 tests verifying ... — committed to dotnet/runtimelab by joperezr 4 years ago

Most upvoted comments

My reservation about increasing the timeout is that it potentially increases the time of our PR validation when it’s not clear whether we’re getting value in return. If I added some new Linq tests today, and it caused the full set to take 16 minutes on WASM, I wouldn’t push another change to increase the timeout. I’d scale back the test. Is there any reason to believe that 15 mins isn’t a reasonable quota to test all of Linq innerloop (on a machine that’s not occupied with any other task) – when no other library we have needs so long?

danmoseley on Aug 21, 2020

I took a closer look at the Kusto data between a run that times out and a good run. It seems both have the same amount of records and their durations are ~13 min. I suspect a longer GC plays a part in the runs that time out. Wasm does not have the benefit of a separate GC thread.

I’m asking around if there are some GC knobs to tweak, but I feel like increasing the timeout might actually be the prudent choice.

steveisok on Aug 21, 2020

We should disable and link to this issue while we investigate.

steveisok on Aug 20, 2020