runtime: Perf regressions in Perf_Matrix benchmarks

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	10b2f7c858934c27c56aa13845cb47064bfe87e9
Compare	a842e7a1dc6c241c928c6291411393cdb2516608
Diff	Diff

Regressions in System.IO.Tests.BinaryWriterTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
WriteAsciiChar - Duration of single invocation	3.44 ns	4.71 ns	1.37	0.17	True

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.BinaryWriterTests*'

Payloads

Baseline Compare

Histogram

System.IO.Tests.BinaryWriterTests.WriteAsciiChar

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	10b2f7c858934c27c56aa13845cb47064bfe87e9
Compare	a842e7a1dc6c241c928c6291411393cdb2516608
Diff	Diff

Regressions in System.Numerics.Tests.Perf_Matrix3x2

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
InequalityOperatorBenchmark - Duration of single invocation	6.33 ns	12.21 ns	1.93	0.07	True
LerpBenchmark - Duration of single invocation	8.16 ns	13.30 ns	1.63	0.04	True
NegationOperatorBenchmark - Duration of single invocation	6.35 ns	10.87 ns	1.71	0.10	False
AddBenchmark - Duration of single invocation	8.96 ns	15.89 ns	1.77	0.08	True

graph graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_Matrix3x2*'

Payloads

Baseline Compare

Histogram

System.Numerics.Tests.Perf_Matrix3x2.InequalityOperatorBenchmark

System.Numerics.Tests.Perf_Matrix3x2.LerpBenchmark

System.Numerics.Tests.Perf_Matrix3x2.NegationOperatorBenchmark

System.Numerics.Tests.Perf_Matrix3x2.AddBenchmark

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	10b2f7c858934c27c56aa13845cb47064bfe87e9
Compare	a842e7a1dc6c241c928c6291411393cdb2516608
Diff	Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<Nullable<DateTimeOffset>>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
DeserializeFromUtf8Bytes - Duration of single invocation	130.22 ns	140.77 ns	1.08	0.05	True

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;*'

Payloads

Baseline Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<Nullable<DateTimeOffset>>.DeserializeFromUtf8Bytes

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	10b2f7c858934c27c56aa13845cb47064bfe87e9
Compare	a842e7a1dc6c241c928c6291411393cdb2516608
Diff	Diff

Regressions in System.Collections.Sort<BigStruct>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
Array_ComparerStruct - Duration of single invocation	33.94 μs	46.92 μs	1.38	0.02	True
Array_Comparison - Duration of single invocation	30.67 μs	45.16 μs	1.47	0.03	True
Array_ComparerClass - Duration of single invocation	31.22 μs	44.95 μs	1.44	0.07	True

graph graph graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Sort&lt;BigStruct&gt;*'

Payloads

Baseline Compare

Histogram

System.Collections.Sort<BigStruct>.Array_ComparerStruct(Size: 512)

System.Collections.Sort<BigStruct>.Array_Comparison(Size: 512)

System.Collections.Sort<BigStruct>.Array_ComparerClass(Size: 512)

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	10b2f7c858934c27c56aa13845cb47064bfe87e9
Compare	a842e7a1dc6c241c928c6291411393cdb2516608
Diff	Diff

Regressions in System.Collections.IterateForEachNonGeneric<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Queue - Duration of single invocation	3.61 μs	3.85 μs	1.07	0.06	False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateForEachNonGeneric&lt;Int32&gt;*'

Payloads

Baseline Compare

Histogram

System.Collections.IterateForEachNonGeneric<Int32>.Queue(Size: 512)

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	10b2f7c858934c27c56aa13845cb47064bfe87e9
Compare	a842e7a1dc6c241c928c6291411393cdb2516608
Diff	Diff

Regressions in System.Collections.IterateForEach<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ConcurrentStack - Duration of single invocation	2.49 μs	2.69 μs	1.08	0.01	False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateForEach&lt;Int32&gt;*'

Payloads

Baseline Compare

Histogram

System.Collections.IterateForEach<Int32>.ConcurrentStack(Size: 512)

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	10b2f7c858934c27c56aa13845cb47064bfe87e9
Compare	a842e7a1dc6c241c928c6291411393cdb2516608
Diff	Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ImmutableSortedSet - Duration of single invocation	28.60 μs	31.25 μs	1.09	0.00	True

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline Compare

Histogram

System.Collections.ContainsFalse<Int32>.ImmutableSortedSet(Size: 512)

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	10b2f7c858934c27c56aa13845cb47064bfe87e9
Compare	a842e7a1dc6c241c928c6291411393cdb2516608
Diff	Diff

Regressions in System.Numerics.Tests.Perf_Vector2

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
TransformNormalByMatrix3x2Benchmark - Duration of single invocation	1.99 ns	8.09 ns	4.06	0.14	True

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_Vector2*'

Payloads

Baseline Compare

Histogram

System.Numerics.Tests.Perf_Vector2.TransformNormalByMatrix3x2Benchmark

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	10b2f7c858934c27c56aa13845cb47064bfe87e9
Compare	a842e7a1dc6c241c928c6291411393cdb2516608
Diff	Diff

Regressions in System.Memory.Span<Byte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SequenceCompareTo - Duration of single invocation	9.92 ns	12.15 ns	1.23	0.08	False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

Payloads

Baseline Compare

Histogram

System.Memory.Span<Byte>.SequenceCompareTo(Size: 512)

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	10b2f7c858934c27c56aa13845cb47064bfe87e9
Compare	a842e7a1dc6c241c928c6291411393cdb2516608
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Uri_IsMatch - Duration of single invocation	132.85 ns	143.51 ns	1.08	0.04	False

graph Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled)

Docs

Profiling workflow for dotnet/runtime repository Benchmarking workflow for dotnet/runtime repository

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 21 (20 by maintainers)

Most upvoted comments

Thanks for promptly looking at this.

The only case that isn’t fixed by this is the System.Collections.IterateForEach<Int32>.ConcurrentStack scenario. I’m not sure that #55604 was the cause for this regression.

I see that there is noise because the regression seems to have recovered. Don’t worry about this benchmark.

kunalspathak on Sep 22, 2021