nx: Running job that hits cache and has cached outputs slowed down after upgrading to 16.9.0
Current Behavior
Hi!
In our monorepo we have build jobs that produce output files. When running build jobs with command nx run-many --target build, and every job hits local cache, the performance profile obtained following instructions from here looks as follows:
In total, build takes ~900 ms. Based on my testing we get similar numbers starting from nx version 16.9.0 and reproduces also in 17.0.0-beta.6.
Expected Behavior
Previously, the job was significantly faster under similar conditions, where every build job got local cache hit. Here is the same example when using nx version 16.8.1.
In total, build took ~20 ms. Therefore, the time it takes to complete job when getting a cache hit has increased by an order of magnitude.
GitHub Repo
No response
Steps to Reproduce
- Benchmark the time it takes to complete a job that hits cache and has cached output when nx version 16.8.1 is installed
- Repeat 1. with nx version 16.9.0 and observe the difference
Nx Report
> NX Report complete - copy this into the issue template
Node : 18.15.0
OS : linux-x64
npm : 9.5.0
nx : 16.9.0
@nx/workspace : 16.5.5
@nx/devkit : 16.8.1
@nrwl/tao : 16.6.0
typescript : 5.1.6
---------------------------------------
Local workspace plugins:
@XXX/local-nx-plugins
---------------------------------------
The following packages should match the installed version of nx
- @nx/workspace@16.5.5
- @nrwl/workspace@16.5.5
- @nx/devkit@16.8.1
- @nrwl/devkit@16.8.1
- @nrwl/tao@16.6.0
To fix this, run `nx migrate nx@16.9.0`
Failure Logs
No response
Package Manager Version
No response
Operating System
- macOS
- Linux
- Windows
- Other (Please specify)
Additional Information
We have observed this slowing down only in jobs that produce output files that are stored to cache.
My colleague - a MAC user, I use linux - reported even bigger slowing down in similar setting. For him running the build with every individual build job hitting local cache may take up to 10 seconds.
About this issue
- Original URL
- State: open
- Created 8 months ago
- Reactions: 4
- Comments: 35 (12 by maintainers)
Also seeing very slow cache restore times. A monorepo with with 100+ projects takes well over 30 seconds to run through a full suite of cache restores. With
NX_DAEMON=falseit takes less than two seconds.Same issue here.
NX_DAEMON=true– 128 seconds to run build across all projectsNX_DAMEON=false– down to 22.83 seconds.Notable difference in output.
NX_DAEMON=true, all task runs report as[existing outputs match the cache, left as is].NX_DAEMON=false, most task runs report as[local cache]What is the difference between local cache and existing outputs match the cache? I assume local cache is a cache hit based on matching hash, and existing outputs means the tasks actually ran, but the outputs are identical. I guess the only time you are saving is file writes in the latter scenario? In the former, no compilation takes place which is the majority of the cpu cycles being eaten.
Average number of files in
dist/is 27. The distribution is:Here are the new logs with Nx 16.9.0:
👍 the issues could be related. Let’s see if Cammisuli or someone else from Nx has further advise and if these two issues should be split to separate github issues or not
I understand what you mean, but my problem here is that when I turn on / off the daemon, with all the caches are hit. Why is the running speed much slower when the daemon process is turned on? This question is the same as this origin issue.
The cpuprofile shows there are multiple idle when daemon process is turned on.
daemon off
daemon on
Since daemon on is the default beheviour, I wonder if this meets your expect? I know the daemon will spawn some processes to watch on the file changes, but now this optimization for large projects seems to have slowed down the running speed of small and medium-sized projects a lot.
I could have been more clear. I meant the output configurations in
nx.json. Here are the results I get:Nx 16.8.1
nx run-many --target=type-check --output-style=statictakes 2.0 seconds with configurationAnd,
nx run-many --target=type-check --output-style=statictakes 2.0 seconds with configurationThe execution time is the same with full cache hit.
Nx 16.9.0
nx run-many --target=type-check --output-style=statictakes 31.2 seconds (!) with configurationAnd,
nx run-many --target=type-check --output-style=statictakes 2.0 seconds with configurationHaving the
outputsdefined takes 10x more time!du -sh node_modules/.cache/nxsays 4.6M so the size of the output files should not be the problem.@Cammisuli if it helps, I can drop few
console.timestatements somewhere innode_modules/nxto help pin point the problem area. Just let me know where