runtime: System.Numerics.Vectors.Tests tests crash on Red Hat CI machines

We do a daily build+test run of dotnet/runtime on Fedora 32 and RHEL8. Since Aug 6th, System.Numerics.Vectors.Tests is reporting test failures and crashes with SIGSEGV.

CI output:

  pushd /home/tester/runtime/artifacts/bin/System.Numerics.Vectors.Tests/net5.0-Debug
  /home/tester/runtime/artifacts/bin/testhost/net5.0-Linux-Debug-x64/dotnet exec --runtimeconfig System.Numerics.Vectors.Tests.runtimeconfig.json --depsfile System.Numerics.Vectors.Tests.deps.json xunit.console.dll System.Numerics.Vectors.Tests.dll -xml testResults.xml -nologo -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
  popd
  ===========================================================================================================
  ~/runtime/artifacts/bin/System.Numerics.Vectors.Tests/net5.0-Debug ~/runtime/src/libraries/System.Numerics.Vectors/tests
  [37m  Discovering: System.Numerics.Vectors.Tests (method display = ClassAndMethod, method display options = None)
  [m[37m  Discovered:  System.Numerics.Vectors.Tests (found 1247 test cases)
  [m[37m  Starting:    System.Numerics.Vectors.Tests (parallel test collections = on, max threads = 4)
      System.Numerics.Tests.GenericVectorTests.LessThanAllInt16 [FAIL]
  [m[31;1m[m[37m      Assert.True() Failure
  [m[37m      Expected: True
  [m[37m      Actual:   False
  [m[30;1m      Stack Trace:
  [m[37m        /home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/GenericVectorTests.cs(1849,0): at System.Numerics.Tests.GenericVectorTests.TestVectorLessThanAll[T]()
  [m[37m        /home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/GenericVectorTests.cs(1819,0): at System.Numerics.Tests.GenericVectorTests.LessThanAllInt16()
  [m[31;1m[m[37m      Assert.True() Failure
  [m[37m      Expected: True
  [m[37m      Actual:   False
  [m[30;1m      Stack Trace:
  [m[37m        /home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/GenericVectorTests.cs(1582,0): at System.Numerics.Tests.GenericVectorTests.TestVectorGreaterThanAll[T]()
  [m[37m        /home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/GenericVectorTests.cs(1544,0): at System.Numerics.Tests.GenericVectorTests.GreaterThanAllInt16()
      System.Numerics.Tests.GenericVectorTests.GreaterThanAllInt16 [FAIL]
  [m[31;1m[m[37m      Assert.False() Failure
  [m[37m      Expected: False
  [m[37m      Actual:   True
  [m[30;1m      Stack Trace:
  [m[37m        /home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/GenericVectorTests.cs(1808,0): at System.Numerics.Tests.GenericVectorTests.TestVectorLessThanAny[T]()
  [m[37m        /home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/GenericVectorTests.cs(1794,0): at System.Numerics.Tests.GenericVectorTests.LessThanAnySingle()
      System.Numerics.Tests.GenericVectorTests.LessThanAnySingle [FAIL]
  [m[31;1m[m[37m      Assert.Equal() Failure
  [m[37m      Expected: 4294967295
  [m[37m      Actual:   2574167081
  [m[30;1m      Stack Trace:
  [m[37m        /home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/GenericVectorTests.cs(1073,0): at System.Numerics.Tests.GenericVectorTests.<>c__DisplayClass329_0`1.<TestSubtractionOverflow>b__0(Int32 index, T val)
  [m[37m        /home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/GenericVectorTests.cs(3138,0): at System.Numerics.Tests.GenericVectorTests.ValidateVector[T](Vector`1 vector, Action`2 indexValidationFunc)
  [m[37m        /home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/GenericVectorTests.cs(1070,0): at System.Numerics.Tests.GenericVectorTests.TestSubtractionOverflow[T]()
  [m[37m        /home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/GenericVectorTests.cs(1055,0): at System.Numerics.Tests.GenericVectorTests.SubtractionOverflowUInt32()
      System.Numerics.Tests.GenericVectorTests.SubtractionOverflowUInt32 [FAIL]
  [m[31;1m[m[37m      Assert.True() Failure
  [m[37m      Expected: True
  [m[37m      Actual:   False
  [m[30;1m      Stack Trace:
  [m[37m        /home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/GenericVectorTests.cs(2098,0): at System.Numerics.Tests.GenericVectorTests.TestVectorEqualsAll[T]()
  [m[37m        /home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/GenericVectorTests.cs(2071,0): at System.Numerics.Tests.GenericVectorTests.VectorEqualsAllInt32()
      System.Numerics.Tests.GenericVectorTests.VectorEqualsAllInt32 [FAIL]
  /home/tester/runtime/artifacts/bin/System.Numerics.Vectors.Tests/net5.0-Debug/RunTests.sh: line 161: 43541 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Numerics.Vectors.Tests.runtimeconfig.json --depsfile System.Numerics.Vectors.Tests.deps.json xunit.console.dll System.Numerics.Vectors.Tests.dll -xml testResults.xml -nologo -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
  [m~/runtime/src/libraries/System.Numerics.Vectors/tests
  ----- end Mon Aug 31 08:03:44 UTC 2020 ----- exit code 139 ----------------------------------------------------------
  exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
  Looking around for any Linux dump..
  Looking for files matching core.* ...
/home/tester/runtime/eng/testing/tests.targets(117,5): error : One or more tests failed while running tests from 'System.Numerics.Vectors.Tests'. [/home/tester/runtime/src/libraries/System.Numerics.Vectors/tests/System.Numerics.Vectors.Tests.csproj]

When I run on my development machine, the tests pass without crashing.

cc @omajid @RheaAyase

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 29 (28 by maintainers)

Most upvoted comments

Thanks! Our CI machine is happy now! 🎉 🎉

tmds on Sep 11, 2020

@danmosemsft - I’ve taken a look at the changes in #40167, and found what seems to be a bug. @tannergooding has a fix for that in #42089 and we can see if that fixes it. It might explain an issue in MethodTable::GetBaseSize(), as the issue is in one of the paths for determining AVX support, which impacts the size of the Vector<T> type.

CarolEidt on Sep 10, 2020