BenchmarkDotNet: benchmark / beforeEverything fails when running in docker-win

I am trying to run a benchmark inside a docker-win container using hyper-v. When I run the benchmark with a default config I see an error like:

BeforeEverythingElse
exitcode != 0

no results were gathered

when I run using the debugInProcess config - the benchmark runs.

I tried the advice here: https://benchmarkdotnet.org/articles/guides/troubleshooting.html#in-the-same-process

but the execution never seems to make it to my benchmark to even enter the while loop.

debugging the resulting genereted.exe using MDbg.exe just seem to hang forever at the BeforeAnythingElse step.

Is there anything that would prevent the benchmarks from running in docker-win?

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 16 (9 by maintainers)

Most upvoted comments

Since BenchmarkDotNet is maintained by MS

It is not. From all the maintainers only I work at MS and I contribute to it in my free time (unless there is a bug|feature blocking my Team). It’s part of the .NET Foundation but in reality, it means that we get a logo and CI for free.

I am sorry but I currently don’t have the time to dig into that.

takes to long to run. Prefer to use out-of-process toolchains for long-running benchmarks

To avoid this problem you can create a custom instance of InProcessToolchain and change the default timeout to a bigger value.

https://benchmarkdotnet.org/api/BenchmarkDotNet.Toolchains.InProcess.NoEmit.InProcessNoEmitToolchain.html#BenchmarkDotNet_Toolchains_InProcess_NoEmit_InProcessNoEmitToolchain__ctor_System_TimeSpan_System_Boolean_

using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
using BenchmarkDotNet.Toolchains.InProcess.NoEmit;
using System;

namespace BenchmarkDotNet.Samples
{
    class Program
    {
        static void Main(string[] args) => BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly)
            .Run(args, DefaultConfig.Instance.With(Job.Default.With(new InProcessNoEmitToolchain(timeout: TimeSpan.FromHours(1), logOutput: false))));
    }
}

adamsitnik on Jan 6, 2020

I’m getting this same error while trying to run Bencmarkdotnet on Microsoft Azure DevOps. When I run it using [InProcess], it throws an exception “…takes to long to run. Prefer to use out-of-process toolchains for long-running benchmarks”.

NB: Both InProcess and out of process run successfully on my local PC.

Has anyone gotten around this?

onedami on Jan 2, 2020

I have the same problem. My output is:

// BeforeAnythingElse
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!

And that’s all. At the end of process the benchmark is listed as with issue, but there is no issue information in the log.

melanchall on Aug 7, 2019

Similar problem on Azure DevOps for net48 (same for net461) on windows-2019.

With

Get-EventLog -LogName Application -Newest 5

I can see:

MachineName        : fv-az659
Data               : {}
Index              : 8267
Category           : (0)
CategoryNumber     : 0
EventID            : 1026
EntryType          : Error
Message            : Application: DefaultJob.exe
                     Framework Version: v4.0.30319
                     Description: The process was terminated due to an unhandled exception.
                     Exception Info: System.NotSupportedException
                        at BenchmarkDotNet.Engines.ConsoleHost.SendSignal(BenchmarkDotNet.Engines.HostSignal)
                        at 
                     BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(System.String[])
                        at BenchmarkDotNet.Autogenerated.UniqueProgramName.Main(System.String[])
                     
                     
Source             : .NET Runtime
ReplacementStrings : {Application: DefaultJob.exe
                     Framework Version: v4.0.30319
                     Description: The process was terminated due to an unhandled exception.
                     Exception Info: System.NotSupportedException
UserName           : 
Site               : 
Container          :

So it narrows down to https://github.com/dotnet/BenchmarkDotNet/blob/3223c94a92050147a02482a3810860703f0c5171/src/BenchmarkDotNet/Engines/ConsoleHost.cs#L25-L35

On my local dev-machine (win-10) there’s no problem. Also there’s no problem for .NET Core targets.

gfoidl on Mar 24, 2020

The issue is opened more than half a year ago and still no fix. Since BenchmarkDotNet is maintained by MS I suppose it’s quite easy to diagnose the problem on Azure Pipelines and solve the issue.

melanchall on Jan 3, 2020