argo-cd: High CPU usage in repo server for plugin detection with >8.000 apps

Checklist:

I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
I’ve included steps to reproduce the bug.
I’ve pasted the output of argocd version.

Describe the bug

Using 2.8.3 of argocd we can see high cpu usages in the repo server for detecting the plugins.

We are using a huge monorepo for our applications, without any templating (just plain yaml). But the detection of plugins take a significant amount of time.

Flame graph with pixie: Bildschirmfoto 2023-09-08 um 13 47 16

Bildschirmfoto 2023-09-08 um 13 46 59

Another one on cleanup: Bildschirmfoto 2023-09-08 um 12 57 08

Slack discussion: https://cloud-native.slack.com/archives/C01TSERG0KZ/p1694514516286809?thread_ts=1694175483.721089&cid=C01TSERG0KZ

CC: @csantanapr

To Reproduce

Apply thousand of apps at the same time

Expected behavior

Apply them “fast”

Screenshots

Version

v2.8.3+77556d9

About this issue

Original URL
State: open
Created 9 months ago
Reactions: 2
Comments: 20 (8 by maintainers)

Most upvoted comments

@JuozasVainauskas Argo CD only does plugin “discovery” if you haven’t explicitly specified in your App manifest that you want something besides a plugin. For example:

kind: Application
spec:
  source:
    kustomize:
      images: [a=b]

For this app, Argo CD would skip plugin discovery because it automatically knows it’ll be using Kustomize instead.

crenshaw-dev on Oct 3, 2023

Update: we have successfully solved the performance issue by setting --plugin-tar-exclude value to .git/* and migrated argocd-cm plugins to sidecars.

JuozasVainauskas on Oct 11, 2023

Thanks @crenshaw-dev the repo only consists of yaml files, but I still use it to exclude the .git folder.

I also experience that I have to lower the parallel repo actions from 50 to 5 otherwise I’ll end up in a strange deadlock situation. Could be because of the plugin detect, too.

woehrl01 on Oct 3, 2023

No actually, it’s the same commit, but I have a mono repo, so it does the resolving 8.000 times for each root folder of the apps.

Great, I’ll check the directory part

woehrl01 on Oct 2, 2023