azure-pipelines-agent: NuGet restore is very slow on self-hosted agent run as service

Agent Version and Platform

Current agent version: ‘2.154.3’ OS: Windows Platform: x64

Azure DevOps Type and Version

dev.azure.com, just build agent is self-hosted. https://dev.azure.com/krossk/

What’s not working?

NuGet restore task is very slow. I have already tried Developer Community. So the setup is this:

We use latest nuget.exe, now it is the version 5.1.0.
We use custom nuget.config where there are two nuget sources: standard nuget.org and our own nuget server. Our nuget server is not private. It is public, so from the agent view, it is the same as nuget.org, just different URL.
Solution has 7 projects.
The total number of restored packages is 81. Only 4 of them are hosted on our own nuget server, the rest are packages from nuget.org.
Total size of restored packages is around 830 MB.
The build runs on our self-hosted machine/agent.
Our build agent(s) run as a service.
The machine is not bad: Intel Xeon, 64GB RAM, SSD disk. And this is our build machine - there is no other software running.

NuGet restore task always takes around 11 minutes (usually between 10 and 13). And all the packages are in the local NuGet cache, they are not downloaded from the internet. I tried standard NuGetCommand@2 task and also simple script step running nuget.exe directly - this makes no difference.

At the Developer Community we were suggested to try changing the mode the agent is running. And this is the thing. When the agent is started manually (interactively), everything is OK. NuGet restore takes 15 seconds when all the packages are in local cache, and around 40 seconds when the cache is empty and the packages are downloaded from the internet.

When the agent is started as a service again, package restore lasts 11 minutes again.

We do not experience this slowness with the other types of steps (build, publish artifacts…). All of them are fine, just nuget restore is not working as expected.

For testing purposes, there were no steps in the build pipeline, just NuGet restore.

Agent and Worker’s Diagnostic Logs

Agent_20190719-140904-utc.log Worker_20190722-140600-utc.log

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 19
Comments: 17 (3 by maintainers)

Most upvoted comments

I’ve noticed that NuGet restore is also very slow on Microsoft-hosted agents (also see #2134, which got dismissed). I guess the underlying issue is the same.

Anyway, here’s some info I found out from a project of mine, in case it could be helpful:

Caching the global packages cache does not seem to help, so the problem is apparently not tied to network speed. This is slow even though NuGet is not supposed to need network access.
I took a binary log of a restore through MSBuild (with all packages already cached), and here’s what it shows:
- Most time is spent in RestoreTask, and it is definitely the bottleneck
- The MsBuild task which invokes the _IsProjectRestoreSupported target (in _FilterRestoreGraphProjectInputItems) also seems a bit too slow, but this may be an unrelated issue.
- The logs of RestoreTask are similar to what I’m getting on my machine, and I couldn’t spot any meaningful difference except for the concurrent job count. Unfortunately, MSBuildLogViewer doesn’t show the timestamps of individual log lines and I don’t know if this information is available in the binlog itself, but I can check that.

ltrzesniewski on Feb 12, 2020

We also experience this issue! Some time 1 month back both Jenkins and Azure DevOps Server, nuget restore times went x10. For the Jenkins we run the service as a user, I just logged in as that user and restarted the server. After that the problem was solved!

For Azure DevOps ServerI have tried more or less everything I can think of! The times are not going down 😕

Setting NUGET_CERT_REVOCATION_MODE = “offline” solved the problem, as the server will not use “internet”. Download only from internal nuget storage…

ArtourK on Oct 27, 2020

For me, the restore took 3 minutes. Ended up switching from NuGet restore task (NuGetCommand@2) to DotNet restore task (DotNetCoreCLI@2). Now the restore takes only 16 seconds.

janisBerz on Feb 3, 2022

@anatolybolshakov Can you please share how to set this “environment variable” in Azure Pipelines? The documentation I found does not say anything about setting environment variables… https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#environment-variables

DerekFoulk on Jul 16, 2021

Hi everyone, this seems to be a known issue with NuGet - please see this link for more details. As @ArtourK mentioned - one of possible solution could be to set environment variable NUGET_CERT_REVOCATION_MODE = “offline” - it is not recommended under normal circumstances, but could help to resolve issue with longer execution time. Let me close this at the moment since it doesn’t seem to be related to agent - please feel free to ask any further questions.

anatolybolshakov on Jan 19, 2021