argo-cd: High CPU usage in repo server for plugin detection with >8.000 apps
Checklist:
- I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
- I’ve included steps to reproduce the bug.
- I’ve pasted the output of
argocd version
.
Describe the bug
Using 2.8.3 of argocd we can see high cpu usages in the repo server for detecting the plugins.
We are using a huge monorepo for our applications, without any templating (just plain yaml). But the detection of plugins take a significant amount of time.
Flame graph with pixie:
Another one on cleanup:
Slack discussion: https://cloud-native.slack.com/archives/C01TSERG0KZ/p1694514516286809?thread_ts=1694175483.721089&cid=C01TSERG0KZ
CC: @csantanapr
To Reproduce
Apply thousand of apps at the same time
Expected behavior
Apply them “fast”
Screenshots
Version
v2.8.3+77556d9
About this issue
- Original URL
- State: open
- Created 9 months ago
- Reactions: 2
- Comments: 20 (8 by maintainers)
@JuozasVainauskas Argo CD only does plugin “discovery” if you haven’t explicitly specified in your App manifest that you want something besides a plugin. For example:
For this app, Argo CD would skip plugin discovery because it automatically knows it’ll be using Kustomize instead.
Update: we have successfully solved the performance issue by setting
--plugin-tar-exclude
value to.git/*
and migrated argocd-cm plugins to sidecars.Thanks @crenshaw-dev the repo only consists of yaml files, but I still use it to exclude the .git folder.
I also experience that I have to lower the parallel repo actions from 50 to 5 otherwise I’ll end up in a strange deadlock situation. Could be because of the plugin detect, too.
No actually, it’s the same commit, but I have a mono repo, so it does the resolving 8.000 times for each root folder of the apps.
Great, I’ll check the directory part