BenchmarkDotNet: Possible Bug with InProcessAttribute

Hey there BenchmarkDotNet Community and Team,

Please see a very simple project I have created here: https://github.com/Mike-EEE/BenchmarkDotNet.InProcess/

I am encountering a weird issue that occurs when the InProcess attribute is used, and when the BenchmarkDotNet.Artifacts folder exists.

On my non-VM (host machine), when I run the above benchmark for the first time and BenchmarkDotNet.Artifacts does not exist or if I run it subsequent times while deleting this folder beforehand, I get results such as this:

  Method |     Mean |     Error |    StdDev |
-------- |---------:|----------:|----------:|
 ToArray | 18.61 us | 0.1818 us | 0.1518 us |

When BenchmarkDotNet.Artifacts does exist, I get the following results consistently:

  Method |     Mean |     Error |    StdDev |
-------- |---------:|----------:|----------:|
 ToArray | 20.64 us | 0.1354 us | 0.1267 us |

Please note the 2us difference.

On my VM (I know there are differences in results, but work with me here), I get the opposite behavior. On the first run when BenchmarkDotNet.Artifacts does not exist, I get the following results:

  Method |     Mean |     Error |    StdDev |
-------- |---------:|----------:|----------:|
 ToArray | 21.55 us | 0.4923 us | 0.5268 us |

When BenchmarkDotNet.Artifacts does exist, I get the following results:

  Method |     Mean |     Error |    StdDev |
-------- |---------:|----------:|----------:|
 ToArray | 19.13 us | 0.3791 us | 0.3361 us |

To add to the mix here, if I comment out this line, I get the 19.xx consistently every time on my VM, regardless of BenchmarkDotNet.Artifacts exists. This line does not impact my non-VM host machine.

Please also note that running this benchmark w/o the InProcess attribute results in a consistent 19.xx result on my VM, regardless on if I comment out the offending line above or not. So it would seem the quirkiness I am experiencing here has to do with running in-process.

To add more context on the difference between my VM and host (non-VM machine), I am performing a “Publish…” from the right-click of the project in the above solution, and then copying/pasting the contents of this to a folder on my host (non-VM) machine. If there is another accepted/expected way of publishing BDN contents to another machine, this would be great to know. Otherwise, the publish profile has been added to the solution above for your convenience.

I hope all of this makes sense. Please let me know if there is something I have misunderstood here and/or if you have any questions. For my side, it would be nice to know if is it considered good or bad practice to “seed” the calls to the benchmarks as outlined in the differentiating line above? That would help clarify things, but outside of that I feel there is some inconsistent behavior here that might be due to a latent bug and/or further misunderstanding on my part.

Thank you for any assistance/guidance you can provide!