runtime: Regressions from Roslyn update

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8549002b2df6bf64c7c882d8f117630beba762d
Compare 8a2b72e3140b14a2ec915345b2747eecdecff943
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetHashCodeBenchmark - Duration of single invocation 79.04 ns 105.44 ns 1.33 0.06 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*'

Payloads

Baseline Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Byte>.GetHashCodeBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 105.4351294570456 > 83.15154770429102.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -38.87873117596192 (T) = (0 -105.35273354069511) / Math.Sqrt((3.992587969600447 / (23)) + (7.861876182668589 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.31787460021963265 = (79.94139466921767 - 105.35273354069511) / 79.94139466921767 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline a8549002b2df6bf64c7c882d8f117630beba762d
Compare 8a2b72e3140b14a2ec915345b2747eecdecff943
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetHashCodeBenchmark - Duration of single invocation 39.22 ns 51.70 ns 1.32 0.05 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*'

Payloads

Baseline Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt16>.GetHashCodeBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 51.69693024643556 > 41.1064538719314.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -36.663764455094125 (T) = (0 -51.68067070574212) / Math.Sqrt((1.65991615061608 / (22)) + (1.9274338623056948 / (31))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (31) - 2, .025) and -0.35719061117778034 = (38.07915430603609 - 51.68067070574212) / 38.07915430603609 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8549002b2df6bf64c7c882d8f117630beba762d
Compare 8a2b72e3140b14a2ec915345b2747eecdecff943
Diff Diff

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetHashCodeBenchmark - Duration of single invocation 9.48 ns 12.98 ns 1.37 0.07 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*'

Payloads

Baseline Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.GetHashCodeBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.97857567164073 > 9.912366475799981.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -93.87222615864435 (T) = (0 -13.031390688994902) / Math.Sqrt((0.011639596703320161 / (23)) + (0.027995569909519624 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.3706358320006545 = (9.50755144783686 - 13.031390688994902) / 9.50755144783686 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8549002b2df6bf64c7c882d8f117630beba762d
Compare 8a2b72e3140b14a2ec915345b2747eecdecff943
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Double>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetHashCodeBenchmark - Duration of single invocation 13.38 ns 14.95 ns 1.12 0.03 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*'

Payloads

Baseline Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Double>.GetHashCodeBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.948886232048702 > 14.071781964405073.
IsChangePoint: Marked as a change because one of 11/2/2022 4:25:40 PM, 12/14/2022 5:20:21 PM, 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -23.26476081715157 (T) = (0 -14.987631064522386) / Math.Sqrt((0.05384720602955832 / (23)) + (0.045076784078112886 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.10574021657368117 = (13.554387223939488 - 14.987631064522386) / 13.554387223939488 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8549002b2df6bf64c7c882d8f117630beba762d
Compare 8a2b72e3140b14a2ec915345b2747eecdecff943
Diff Diff

Regressions in System.Tests.Perf_HashCode

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Combine_7 - Duration of single invocation 96.03 μs 105.47 μs 1.10 0.00 False
Combine_2 - Duration of single invocation 56.55 μs 63.40 μs 1.12 0.00 False
Combine_1 - Duration of single invocation 46.17 μs 52.16 μs 1.13 0.00 False
Combine_5 - Duration of single invocation 74.27 μs 82.59 μs 1.11 0.00 False
Combine_4 - Duration of single invocation 63.99 μs 71.89 μs 1.12 0.00 False
Combine_6 - Duration of single invocation 82.93 μs 93.13 μs 1.12 0.00 False
Combine_8 - Duration of single invocation 82.16 μs 90.71 μs 1.10 0.00 False
Combine_3 - Duration of single invocation 69.56 μs 74.71 μs 1.07 0.00 False

graph graph graph graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_HashCode*'

Payloads

Baseline Compare

Histogram

System.Tests.Perf_HashCode.Combine_7


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 105.4663142677373 > 100.81468016679449.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -161.18619752295376 (T) = (0 -105545.67733509371) / Math.Sqrt((6585.849273667598 / (23)) + (98379.09306130768 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.09869467137568116 = (96064.61202086239 - 105545.67733509371) / 96064.61202086239 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_HashCode.Combine_2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.40146640519568 > 59.350448427346585.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -521.8008790934781 (T) = (0 -63448.79334557399) / Math.Sqrt((1550.9166089246503 / (23)) + (3334.649210823573 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.12207380790148624 = (56546.006954958306 - 63448.79334557399) / 56546.006954958306 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_HashCode.Combine_1

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 52.161184375000005 > 48.49616808904867.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -474.7956288486027 (T) = (0 -52200.563625643365) / Math.Sqrt((2370.9048049895378 / (22)) + (1632.5099591971323 / (31))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (31) - 2, .025) and -0.13020631808142258 = (46186.756161703495 - 52200.563625643365) / 46186.756161703495 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_HashCode.Combine_5

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 82.59137885338346 > 77.98060776806872.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -73.22615233862166 (T) = (0 -82505.62178616505) / Math.Sqrt((104660.75114867605 / (23)) + (244552.67765220546 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.10986241775819691 = (74338.6031151659 - 82505.62178616505) / 74338.6031151659 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_HashCode.Combine_4

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 71.89388650884665 > 67.18613039001559.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -281.6459962191387 (T) = (0 -71670.10661699313) / Math.Sqrt((709.6575365608301 / (23)) + (22155.12236267368 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.12019719537860528 = (63979.90185359284 - 71670.10661699313) / 63979.90185359284 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_HashCode.Combine_6

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 93.12999140625 > 87.02870333416007.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -115.48185935902607 (T) = (0 -93069.88297768412) / Math.Sqrt((3333.509569601689 / (23)) + (235566.862937844 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.12257404511481351 = (82907.56710677843 - 93069.88297768412) / 82907.56710677843 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_HashCode.Combine_8

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 90.71372703516377 > 86.27665725675548.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -535.3374364970878 (T) = (0 -90757.96576528928) / Math.Sqrt((3413.288835681361 / (23)) + (3346.5812596985775 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.10429181728794486 = (82186.57817114303 - 90757.96576528928) / 82186.57817114303 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_HashCode.Combine_3

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 74.7060919059011 > 73.07507285416666.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -140.11060272122677 (T) = (0 -74609.6146257151) / Math.Sqrt((1174.072760930738 / (23)) + (38213.510624717346 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.07213825342420208 = (69589.54629911434 - 74609.6146257151) / 69589.54629911434 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8549002b2df6bf64c7c882d8f117630beba762d
Compare 8a2b72e3140b14a2ec915345b2747eecdecff943
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetHashCodeBenchmark - Duration of single invocation 10.81 ns 13.50 ns 1.25 0.03 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*'

Payloads

Baseline Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int64>.GetHashCodeBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.501432980336178 > 11.349420695138868.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -53.84020803276331 (T) = (0 -13.53650726544601) / Math.Sqrt((0.008029975980635195 / (24)) + (0.06952606554272087 / (31))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (31) - 2, .025) and -0.2530132020379094 = (10.803164119444345 - 13.53650726544601) / 10.803164119444345 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8549002b2df6bf64c7c882d8f117630beba762d
Compare 8a2b72e3140b14a2ec915345b2747eecdecff943
Diff Diff

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetHashCodeBenchmark - Duration of single invocation 38.33 ns 54.55 ns 1.42 0.02 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*'

Payloads

Baseline Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.GetHashCodeBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 54.54619767782453 > 40.21929684002488.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -82.21235737703599 (T) = (0 -54.1115179051457) / Math.Sqrt((0.06252101293171532 / (24)) + (1.0797309196087996 / (31))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (31) - 2, .025) and -0.4163482868152744 = (38.20495171199592 - 54.1115179051457) / 38.20495171199592 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8549002b2df6bf64c7c882d8f117630beba762d
Compare 8a2b72e3140b14a2ec915345b2747eecdecff943
Diff Diff

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetHashCodeBenchmark - Duration of single invocation 19.54 ns 24.49 ns 1.25 0.03 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*'

Payloads

Baseline Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.GetHashCodeBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.489060082989262 > 20.74241711124787.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -24.00227051035366 (T) = (0 -24.845609585691953) / Math.Sqrt((0.07136336798950783 / (23)) + (1.2121066048338698 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.2475992129108641 = (19.91473650237632 - 24.845609585691953) / 19.91473650237632 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8549002b2df6bf64c7c882d8f117630beba762d
Compare 8a2b72e3140b14a2ec915345b2747eecdecff943
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetHashCodeBenchmark - Duration of single invocation 20.29 ns 23.33 ns 1.15 0.04 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*'

Payloads

Baseline Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt32>.GetHashCodeBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.331260832845107 > 21.056982512327213.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -25.472944182933528 (T) = (0 -23.645774694088463) / Math.Sqrt((0.1302430376724827 / (23)) + (0.4615801379454354 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.1826468503281096 = (19.993943828225863 - 23.645774694088463) / 19.993943828225863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8549002b2df6bf64c7c882d8f117630beba762d
Compare 8a2b72e3140b14a2ec915345b2747eecdecff943
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetHashCodeBenchmark - Duration of single invocation 10.81 ns 13.54 ns 1.25 0.05 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*'

Payloads

Baseline Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt64>.GetHashCodeBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.536802932034956 > 11.35145753448431.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -8.574082760730478 (T) = (0 -13.506688811973898) / Math.Sqrt((1.7762903502862928 / (23)) + (0.10097248716469229 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.21965381121373115 = (11.074198832316851 - 13.506688811973898) / 11.074198832316851 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8549002b2df6bf64c7c882d8f117630beba762d
Compare 8a2b72e3140b14a2ec915345b2747eecdecff943
Diff Diff

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetHashCodeBenchmark - Duration of single invocation 6.14 ns 8.24 ns 1.34 0.20 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*'

Payloads

Baseline Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.GetHashCodeBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.243436857714752 > 6.447982991414564.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -8.389320644851091 (T) = (0 -8.063495790669068) / Math.Sqrt((0.03524392530140337 / (24)) + (1.7697574079138925 / (31))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (31) - 2, .025) and -0.33647761884757654 = (6.033393808436607 - 8.063495790669068) / 6.033393808436607 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline a8549002b2df6bf64c7c882d8f117630beba762d
Compare 8a2b72e3140b14a2ec915345b2747eecdecff943
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetHashCodeBenchmark - Duration of single invocation 37.85 ns 51.66 ns 1.37 0.07 False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*'

Payloads

Baseline Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int16>.GetHashCodeBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 51.66120195005749 > 39.72074575315014.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 1/17/2023 6:49:05 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 6:49:05 AM.
IsRegressionStdDev: Marked as regression because -39.559854435578245 (T) = (0 -51.881876605118066) / Math.Sqrt((0.8104291630112155 / (23)) + (2.8603275361064218 / (31))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (31) - 2, .025) and -0.37413871248681874 = (37.75592386246504 - 51.881876605118066) / 37.75592386246504 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 21 (19 by maintainers)

Most upvoted comments

Once we have #8887 I think it makes good sense to experiment with using IsReadOnlyAttribute too as an input to whether reverse copy propagation can be done safely.

I think so.

The codegen difference in the screenshot above looks like the result of https://github.com/dotnet/roslyn/pull/65642, related issue in dotnet/runtime is #73606.