BenchmarkDotNet: Can't use latest version of BenchmarkDotNet

I tried to use the latest bdn master (dotnet/BenchmarkDotNet@73f8fd1dc49f34b79afa4ef16e0204b1369fdd1a) here, but trying to run benchmarks fails with:

Unhandled exception. System.InvalidOperationException: Failed to compare two elements in the array.
 ---> System.ArgumentException: At least one object must implement IComparable.
   at System.Collections.Comparer.Compare(Object a, Object b)
   at System.ValueTuple`3.CompareTo(ValueTuple`3 other)
   at System.ValueTuple`3.System.IComparable.CompareTo(Object other)
   at BenchmarkDotNet.Parameters.ParameterComparer.CompareValues(Object x, Object y)
   at BenchmarkDotNet.Parameters.ParameterComparer.Compare(ParameterInstances x, ParameterInstances y)
   at BenchmarkDotNet.Order.DefaultOrderer.BenchmarkComparer.Compare(BenchmarkCase x, BenchmarkCase y)
   at System.Collections.Generic.ArraySortHelper`1.SwapIfGreater(Span`1 keys, Comparison`1 comparer, Int32 i, Int32 j)
   at System.Collections.Generic.ArraySortHelper`1.PickPivotAndPartition(Span`1 keys, Comparison`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.IntroSort(Span`1 keys, Int32 depthLimit, Comparison`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.IntrospectiveSort(Span`1 keys, Comparison`1 comparer)
   at System.Collections.Generic.GenericArraySortHelper`1.Sort(Span`1 keys, IComparer`1 comparer)
   --- End of inner exception stack trace ---
   at System.Collections.Generic.GenericArraySortHelper`1.Sort(Span`1 keys, IComparer`1 comparer)
   at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Collections.Generic.List`1.Sort(Int32 index, Int32 count, IComparer`1 comparer)
   at BenchmarkDotNet.Order.DefaultOrderer.GetExecutionOrder(ImmutableArray`1 benchmarkCases, IEnumerable`1 order)
   at BenchmarkDotNet.Running.BenchmarkConverter.MethodsToBenchmarksWithFullConfig(Type type, MethodInfo[] benchmarkMethods, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkConverter.TypeToBenchmarks(Type type, IConfig config)
   at BenchmarkDotNet.Running.TypeFilter.<>c__DisplayClass1_0.<Filter>b__0(Type type)
   at System.Linq.Enumerable.SelectListIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.ToArray()
   at BenchmarkDotNet.Running.TypeFilter.Filter(IConfig effectiveConfig, IEnumerable`1 types)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config, Boolean askUserForInput)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
   at MicroBenchmarks.Program.Main(String[] args) in /Users/ankj/dev/performance/src/benchmarks/micro/Program.cs:line 43

This seems to be failing for

System.ValueTuple`3[System.Globalization.CultureInfo,System.Globalization.CompareOptions,System.Boolean], (en-US, Ordinal, False)
System.ValueTuple`3[System.Globalization.CultureInfo,System.Globalization.CompareOptions,System.Boolean], (pl-PL, None, False)`

… , because CultureInfo does not implement IComparable.

IIUC, this is a new code path introduced as part of https://github.com/dotnet/BenchmarkDotNet/pull/2304 .

And it fails for https://github.com/dotnet/performance/blob/dd14c7d44444a0211035af8464b743a55b4dd55e/src/benchmarks/micro/libraries/System.Globalization/StringSearch.cs#L46 .

cc @mrahhal @adamsitnik @LoopedBard3

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 18 (13 by maintainers)

Most upvoted comments

Catch the exception and fallback.

I also find that acceptable. Just catch the exact exception type and compare the message to avoid suppressing erroneous exceptions.

System.ArgumentException: At least one object must implement IComparable.

Just be aware that they can have variable number of types and nested ValueTuples.

Going through ITuple might make that easier.