BenchmarkDotNet: Multiple build/publish failure with `--coreRun` toolchain

If we run benchmarks with CoreRun toolchain and --no-build switch we get bechmarks build errors. This PR https://github.com/dotnet/BenchmarkDotNet/pull/1001 fix the issue adding some build/publish attempts https://github.com/dotnet/BenchmarkDotNet/pull/1001/files#diff-83177b9866652943fa5d5bb8f80d847aR89 I found this issue using new performance https://github.com/dotnet/performance repo. To repro: clone https://github.com/dotnet/corefx and build from root

build -release

clone dotnet/performance and run script(CoreRun.exe will be generated by corefx repo build):

d:\git\performance\scripts\benchmarks_ci.py --frameworks netcoreapp3.0 --filter *.Perf_Dictionary.* --corerun D:\git\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --bdn-arguments="--join"

It works and we can see the “attempts” from output

[2019/01/07 20:13:52][INFO] $ pushd "D:\git\performance\src\benchmarks\micro"
[2019/01/07 20:13:52][INFO] $ dotnet run --project MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-restore --no-build -- --coreRun D:\git\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --filter *.Perf_Dictionary.* --join --packages D:\git\performance\artifacts\packages --runtimes netcoreapp3.0
[2019/01/07 20:14:01][INFO] // Validating benchmarks:
[2019/01/07 20:14:01][INFO] // ***** BenchmarkRunner: Start   *****
[2019/01/07 20:14:01][INFO] // ***** Building 1 exe(s) in Parallel: Start   *****
[2019/01/07 20:14:02][INFO] // start dotnet restore  /p:UseSharedCompilation=false in D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\37a25b85-e8c6-4a15-b795-c55e4af09df2
[2019/01/07 20:14:04][INFO] // command took 2.16s and exited with 0
[2019/01/07 20:14:04][INFO] // start dotnet build -c Release  /p:UseSharedCompilation=false in D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\37a25b85-e8c6-4a15-b795-c55e4af09df2
[2019/01/07 20:14:29][INFO] // command took 25.36s and exited with 1
[2019/01/07 20:14:29][INFO] // start dotnet build -c Release  --no-dependencies /p:UseSharedCompilation=false in D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\37a25b85-e8c6-4a15-b795-c55e4af09df2
[2019/01/07 20:14:32][INFO] // command took 3.2s and exited with 0
[2019/01/07 20:14:32][INFO] // start dotnet publish -c Release  /p:UseSharedCompilation=false in D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\37a25b85-e8c6-4a15-b795-c55e4af09df2
[2019/01/07 20:14:44][INFO] // command took 12.04s and exited with 1
[2019/01/07 20:14:44][INFO] // start dotnet publish -c Release  --no-build /p:UseSharedCompilation=false in D:\git\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\37a25b85-e8c6-4a15-b795-c55e4af09df2
[2019/01/07 20:14:46][INFO] // command took 1.53s and exited with 0

I did some investigation here https://github.com/dotnet/performance/issues/209#issuecomment-451759735 Could be useful understand if we can fix the issue without “attempts” saving time.

/cc @adamsitnik @jorive(https://github.com/dotnet/performance/pull/217#discussion_r245769189)

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 2
Comments: 15 (15 by maintainers)

Commits related to this issue

Improve restore, build and publish projects - Fix for #1002 (#1013) * Comment dotnet publish with --no-build * Fix restore, build and publish projects * Remove LaunchSettings.json — committed to dotnet/BenchmarkDotNet by WojciechNagorski 5 years ago

Most upvoted comments

Ok. I reproduced problem 😃

When I used benchmarks_ci.py I got error:

[2019/01/11 12:37:01][INFO] $ dotnet run --project MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-restore --no-build -- --coreRun C:\Work\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Debug-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --filter *.Perf_Dictionary.* --join --packages C:\Work\performance\artifacts\packages --runtimes netcoreapp3.0


[2019/01/11 12:37:10][INFO] // Validating benchmarks:
[2019/01/11 12:37:10][INFO] // ***** BenchmarkRunner: Start   *****
[2019/01/11 12:37:10][INFO] // ***** Building 1 exe(s) in Parallel: Start   *****
[2019/01/11 12:37:14][INFO] // start dotnet restore  /p:UseSharedCompilation=false in C:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7edbf419-fff9-4cd8-9cdd-64b2369c4bcc
[2019/01/11 12:38:00][INFO] // command took 45.54s and exited with 0
[2019/01/11 12:38:00][INFO] // start dotnet build -c Release  /p:UseSharedCompilation=false in C:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7edbf419-fff9-4cd8-9cdd-64b2369c4bcc
[2019/01/11 12:38:29][INFO] // command took 29.49s and exited with 1
[2019/01/11 12:38:29][INFO] // start dotnet build -c Release  --no-dependencies /p:UseSharedCompilation=false in C:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7edbf419-fff9-4cd8-9cdd-64b2369c4bcc
[2019/01/11 12:38:34][INFO] // command took 5.45s and exited with 0
[2019/01/11 12:38:34][INFO] // start dotnet publish -c Release  /p:UseSharedCompilation=false in C:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7edbf419-fff9-4cd8-9cdd-64b2369c4bcc
[2019/01/11 12:38:48][INFO] // command took 13.78s and exited with 1

and now I can reproduse this error without this script. When I type the commands in the order from script, I got ‘attemps’:

cd "C:\Work\performance\src\benchmarks\micro"

dotnet restore MicroBenchmarks.csproj --packages C:\Work\performance\artifacts\packages

dotnet build MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-restore

dotnet run --project MicroBenchmarks.csproj --configuration Release --framework netcoreapp3.0 --no-restore --no-build -- --coreRun C:\Work\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Debug-x64\shared\Microsoft.NETCore.App\9.9.9\CoreRun.exe --filter *.Perf_Dictionary.* --join --packages C:\Work\performance\artifacts\packages --runtimes netcoreapp3.0

// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start dotnet restore  /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7281b0ff-54e4-4e98-9260-668e5a98d63c
// command took 4.02s and exited with 0
// start dotnet build -c Release  /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7281b0ff-54e4-4e98-9260-668e5a98d63c
// command took 18.57s and exited with 1
// start dotnet build -c Release  --no-dependencies /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7281b0ff-54e4-4e98-9260-668e5a98d63c
// command took 5.27s and exited with 0
// start dotnet publish -c Release  /p:UseSharedCompilation=false in c:\Work\performance\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\7281b0ff-54e4-4e98-9260-668e5a98d63c
// command took 13.57s and exited with 1
// ***** Done, took 00:00:43 (43.58 sec)   *****
// Found benchmarks:

I have some Idea how to solve this problem.

WojciechNagorski on Jan 11, 2019

I can help but I need information on what to do it. There #913 (comment) you wrote that dotnet restore --no-dependencies and dotnet build --no-restore --no-dependencies are dengerous. Do you have some idea how to fix this issue or do I have to come up with a solution?

Maybe benchmarkDotNet should have parameters --no-build,–no-dependencies, --no-restore and then pass them to DotNetCliCommand during build?

@wojtpl2 I think that the best option would be to detect that user is running dotnet run with those arguments and re-apply them when building the auto-generated code.

adamsitnik on Jan 9, 2019

@MarcoRossignoli I’ve pushed fix #1013

WojciechNagorski on Jan 11, 2019