runtime: [Perf] Linux/x64: 27 Regressions on 8/17/2022 6:09:21 PM

There are some large wasm regressions here; as much as a 10 x slow down. Unfortunately, there seems to be some kind of glitch in the reporting, and the narrowest I can get the change range is this: https://github.com/dotnet/runtime/compare/543bcc5ee7d6a2b9471b016770227421c43a756e...a250bfe05688352da06c11cbfce33550b388a868.

@lewing @radical @pavelsavara Any likely candidates in that date range?

Automated issue reporting below.

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	543bcc5ee7d6a2b9471b016770227421c43a756e
Compare	f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
Uri_IsNotMatch - Duration of single invocation	444.95 ns	592.31 ns	1.33	0.06	False
Uri_IsMatch - Duration of single invocation	455.72 ns	505.99 ns	1.11	0.05	False
Backtracking - Duration of single invocation	2.45 μs	2.93 μs	1.19	0.10	False
Backtracking - Duration of single invocation	2.45 μs	2.91 μs	1.19	0.10	False
Uri_IsNotMatch - Duration of single invocation	446.25 ns	581.71 ns	1.30	0.05	False
Uri_IsNotMatch - Duration of single invocation	447.94 ns	593.46 ns	1.32	0.04	False

graph graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 592.3093690327008 > 465.23264235782176.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -39.47621223602034 (T) = (0 -594.5364016454982) / Math.Sqrt((78.92038015570309 / (14)) + (73.27831635677934 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.32709221691484786 = (447.99931313563445 - 594.5364016454982) / 447.99931313563445 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 505.9861520887883 > 482.7014638466213.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/25/2022 6:06:34 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -14.329767862712716 (T) = (0 -514.4415933301957) / Math.Sqrt((16.440001607480827 / (14)) + (123.32550738351799 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.12037177742305741 = (459.17043225905917 - 514.4415933301957) / 459.17043225905917 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9266354356553093 > 2.591927353654952.
IsChangePoint: Marked as a change because one of 6/24/2022 6:48:53 PM, 6/29/2022 4:06:39 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -50.14154390540031 (T) = (0 -2934.2504087143566) / Math.Sqrt((556.432351718883 / (14)) + (417.26555557386655 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.18845364681210353 = (2468.964958444245 - 2934.2504087143566) / 2468.964958444245 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9101414399726724 > 2.5780127447204717.
IsChangePoint: Marked as a change because one of 6/24/2022 5:01:29 PM, 6/25/2022 2:40:01 AM, 6/29/2022 4:06:39 AM, 7/12/2022 12:16:14 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -61.70631510275718 (T) = (0 -2923.812224534253) / Math.Sqrt((407.1945204507192 / (14)) + (236.0815117136678 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.18619294374645873 = (2464.870694053968 - 2923.812224534253) / 2464.870694053968 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 581.7136123036038 > 471.5091607783785.
IsChangePoint: Marked as a change because one of 6/25/2022 5:15:47 AM, 6/26/2022 6:11:06 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -65.94725454140149 (T) = (0 -592.9345145576187) / Math.Sqrt((24.613985181656737 / (14)) + (26.562821064295214 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.3181628970020087 = (449.81884705310074 - 592.9345145576187) / 449.81884705310074 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 593.4594763018782 > 471.43001759852376.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/24/2022 6:48:53 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -48.543170908408 (T) = (0 -596.0567739968475) / Math.Sqrt((34.7262092388468 / (14)) + (56.62787404726851 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.3178926565327463 = (452.2802149645615 - 596.0567739968475) / 452.2802149645615 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	543bcc5ee7d6a2b9471b016770227421c43a756e
Compare	f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff	Diff

Regressions in System.Text.Json.Tests.Perf_Reader

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ReadSpanEmptyLoop - Duration of single invocation	92.65 μs	99.48 μs	1.07	0.09	False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation	1.70 μs	1.91 μs	1.13	0.07	False
ReadSpanEmptyLoop - Duration of single invocation	4.25 μs	4.95 μs	1.16	0.09	False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation	11.52 μs	12.59 μs	1.09	0.12	False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation	112.87 μs	118.86 μs	1.05	0.07	False

graph graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Reader*'

Payloads

Baseline Compare

Histogram

System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: Json40KB)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 99.47909983507735 > 97.49529907633277.
IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -9.875672080579108 (T) = (0 -101229.28427961336) / Math.Sqrt((265137.5448871564 / (14)) + (5976092.35420695 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.08769210576340399 = (93067.95897775218 - 101229.28427961336) / 93067.95897775218 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.90914647431983 > 1.777974693072054.
IsChangePoint: Marked as a change because one of 6/22/2022 10:31:12 AM, 7/19/2022 8:53:09 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -24.12292592652499 (T) = (0 -1927.1512790997767) / Math.Sqrt((611.4742780301284 / (14)) + (471.64347753333664 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.1398571366423715 = (1690.6954539728583 - 1927.1512790997767) / 1690.6954539728583 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: LotsOfNumbers)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.946905717995979 > 4.460535892543025.
IsChangePoint: Marked as a change because one of 6/24/2022 6:48:53 PM, 6/25/2022 5:15:47 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -14.376124247128779 (T) = (0 -4757.429574447925) / Math.Sqrt((4758.814727171611 / (14)) + (8231.09975483114 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.11985725605871154 = (4248.246415968665 - 4757.429574447925) / 4248.246415968665 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.58755040658602 > 12.38535184349092.
IsChangePoint: Marked as a change because one of 6/22/2022 10:31:12 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -6.466117999484596 (T) = (0 -12591.240550799725) / Math.Sqrt((69596.82198162597 / (14)) + (79394.98687585884 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06418279234267017 = (11831.840019778581 - 12591.240550799725) / 11831.840019778581 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 118.8626892992424 > 118.55053428507193.
IsChangePoint: Marked as a change because one of 6/17/2022 9:56:33 AM, 7/19/2022 10:33:10 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -7.506023118864004 (T) = (0 -121572.12886191734) / Math.Sqrt((4192742.5510894423 / (14)) + (6482122.752315378 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.06649265512776761 = (113992.4670623754 - 121572.12886191734) / 113992.4670623754 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	543bcc5ee7d6a2b9471b016770227421c43a756e
Compare	f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff	Diff

Regressions in System.Tests.Perf_String

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Format_MultipleArgs - Duration of single invocation	529.02 ns	850.06 ns	1.61	0.14	False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline Compare

Histogram

System.Tests.Perf_String.Format_MultipleArgs

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 850.0557408302284 > 547.6817332098481.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/27/2022 8:54:23 AM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -86.40554379355675 (T) = (0 -848.1242802325862) / Math.Sqrt((88.87602574889881 / (14)) + (69.21223243326581 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.6173858106947179 = (524.3796963127123 - 848.1242802325862) / 524.3796963127123 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	543bcc5ee7d6a2b9471b016770227421c43a756e
Compare	f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff	Diff

Regressions in System.Tests.Perf_Environment

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ExpandEnvironmentVariables - Duration of single invocation	382.06 ns	1.06 μs	2.76	0.11	False
SetEnvironmentVariable - Duration of single invocation	194.79 ns	661.73 ns	3.40	0.16	False

graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Environment*'

Payloads

Baseline Compare

Histogram

System.Tests.Perf_Environment.ExpandEnvironmentVariables

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.05539799053797 > 404.12774324999464.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -100.66329094480135 (T) = (0 -1058.1419394441955) / Math.Sqrt((68.03089949492843 / (14)) + (358.51775965172686 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -1.7472293516620847 = (385.16694603747425 - 1058.1419394441955) / 385.16694603747425 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Environment.SetEnvironmentVariable

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 661.725814258165 > 213.09369271099766.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -164.42300266168527 (T) = (0 -647.4959303229203) / Math.Sqrt((49.2398659816711 / (14)) + (35.11775717258779 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -2.2433183359130027 = (199.6399561379005 - 647.4959303229203) / 199.6399561379005 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	543bcc5ee7d6a2b9471b016770227421c43a756e
Compare	f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff	Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IndexOfString - Duration of single invocation	24.67 ns	36.76 ns	1.49	0.21	False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: “StrIng”, value: “string”, comparisonType: OrdinalIgnoreCase)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.76482912607041 > 26.19735640294146.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -67.5375793028699 (T) = (0 -36.38052307472043) / Math.Sqrt((0.037995025208353364 / (14)) + (0.23599627187678154 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.4615325383200185 = (24.892037721266604 - 36.38052307472043) / 24.892037721266604 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	543bcc5ee7d6a2b9471b016770227421c43a756e
Compare	f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff	Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ICollection - Duration of single invocation	186.65 μs	2.42 ms	12.95	0.01	False
Stack - Duration of single invocation	148.17 μs	2.98 ms	20.11	0.01	False
Queue - Duration of single invocation	186.46 μs	2.42 ms	12.98	0.02	False

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

### Histogram

System.Collections.ContainsFalse<Int32>.ICollection(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4180025503826528 > 196.43006513020836.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -373.14299335258266 (T) = (0 -2427206.700892857) / Math.Sqrt((1083311.6941798208 / (14)) + (323590191.29539 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -11.95486324505923 = (187358.7281454749 - 2427206.700892857) / 187358.7281454749 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.Stack(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9792487988095235 > 155.34581797022406.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -42.64814865820476 (T) = (0 -2532821.408927845) / Math.Sqrt((874624.8610895702 / (14)) + (28136948852.7768 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -16.092038104366907 = (148187.20818792962 - 2532821.408927845) / 148187.20818792962 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4195034313186814 > 195.68580234160373.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -91.92109792877557 (T) = (0 -2470159.070670352) / Math.Sqrt((714329.7028732462 / (14)) + (5552447103.319803 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -12.216076602459486 = (186905.62600179392 - 2470159.070670352) / 186905.62600179392 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	543bcc5ee7d6a2b9471b016770227421c43a756e
Compare	f142128e89b63577a9bbba7e2b760ec82102a7a9
Diff	Diff

Regressions in Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Load - Duration of single invocation	277.65 μs	294.38 μs	1.06	0.09	False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*'

Payloads

Baseline Compare

Histogram

Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: “repeated.xml”)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 294.3815035812672 > 286.28722408634746.
IsChangePoint: Marked as a change because one of 7/1/2022 12:02:05 PM, 7/10/2022 5:59:30 PM, 7/18/2022 3:18:54 PM, 8/5/2022 10:47:59 PM, 8/19/2022 10:17:27 PM falls between 8/5/2022 10:47:59 PM and 8/19/2022 10:17:27 PM.
IsRegressionStdDev: Marked as regression because -9.519909182959154 (T) = (0 -300647.38905054185) / Math.Sqrt((42979330.934151694 / (14)) + (35851042.04585498 / (9))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (9) - 2, .025) and -0.09181732773423353 = (275364.18539395486 - 300647.38905054185) / 275364.18539395486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 21 (19 by maintainers)

Most upvoted comments

We should start with a fix in main and then backport it to rc2.

stephentoub on Sep 16, 2022

Imho such code should be avoided in low level BCL code, its better to have some code duplication.

This isn’t really practical, IMO, for us or our customers who are going to do the same. We have many features that are built on and around generics and generic specialization.

Much like with Vector<T>, Vector64<T>, Vector128<T>, and Vector256<T> this really seems like something that WASM and AOT will have to handle as they are such core concepts to .NET as a whole (especially in perf oriented code).

Many of these algorithms are very long and very complex. It’s not just a small amount of duplication, but large amounts of duplication in some of our most critical code (where having one place to review/update is often core/critical due to its complexity).

tannergooding on Sep 15, 2022