argo-cd: High CPU usage in repo server for plugin detection with >8.000 apps

Checklist:

  • I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • I’ve included steps to reproduce the bug.
  • I’ve pasted the output of argocd version.

Describe the bug

Using 2.8.3 of argocd we can see high cpu usages in the repo server for detecting the plugins.

We are using a huge monorepo for our applications, without any templating (just plain yaml). But the detection of plugins take a significant amount of time.

Flame graph with pixie: Bildschirmfoto 2023-09-08 um 13 47 16

Bildschirmfoto 2023-09-08 um 13 46 59

Another one on cleanup: Bildschirmfoto 2023-09-08 um 12 57 08

Slack discussion: https://cloud-native.slack.com/archives/C01TSERG0KZ/p1694514516286809?thread_ts=1694175483.721089&cid=C01TSERG0KZ

CC: @csantanapr

To Reproduce

Apply thousand of apps at the same time

Expected behavior

Apply them “fast”

Screenshots

Version

v2.8.3+77556d9

About this issue

  • Original URL
  • State: open
  • Created 9 months ago
  • Reactions: 2
  • Comments: 20 (8 by maintainers)

Most upvoted comments

@JuozasVainauskas Argo CD only does plugin “discovery” if you haven’t explicitly specified in your App manifest that you want something besides a plugin. For example:

kind: Application
spec:
  source:
    kustomize:
      images: [a=b]

For this app, Argo CD would skip plugin discovery because it automatically knows it’ll be using Kustomize instead.

Update: we have successfully solved the performance issue by setting --plugin-tar-exclude value to .git/* and migrated argocd-cm plugins to sidecars.

Thanks @crenshaw-dev the repo only consists of yaml files, but I still use it to exclude the .git folder.

I also experience that I have to lower the parallel repo actions from 50 to 5 otherwise I’ll end up in a strange deadlock situation. Could be because of the plugin detect, too.

No actually, it’s the same commit, but I have a mono repo, so it does the resolving 8.000 times for each root folder of the apps.

Great, I’ll check the directory part