argo-cd: repo-cache-expiration is ignored for helm repositories

I’ve asked about this in slack several times over several days and no one has an answer. It doesn’t help that I literally can’t find one example of this flag actually getting used on the entirety of the internet.

Describe the bug

I recently transitioned all my argocd apps to point at a private helm repository we host, instead of pointing the applications back at our gitlab instance. This is because we’re getting customers that need a private installation of our product, not managed by us. This has worked great for the most part, I really like sync’ing to a helm repo versus gitlab.

However, changes to the helm chart aren’t picked up by argo because argo isn’t re-pulling the helm chart from our chart repo. The only way I can get argo to re-pull is to issue a manual hard-refresh. But I can’t get this to happen in any automated fashion. I’ve researched this extensively, and the only thing I could find is this repo-cache-expiration flag that you can set in the repo server command that will expire argo’s redis cache and force it to re-pull the chart from source. I set this flag to 1m for testing. At first I didn’t think I set it correctly, but then I saw that it was working, for the couple of apps I still had argo pointing to gitlab. But for the helm-based applications, this flag seems to be completely ignored. Nothing I’ve been able to do will convince argo to re-pull from our helm chart repo.

To Reproduce

Deploy an app in argocd that syncs to a helm chart hosted in an external repository
Make and commit a change to the chart, in-place (don’t re-version it)
You’ll notice that argocd won’t pick up the change until you hard-refresh

Expected behavior

With the repo-cache-expiration flag set to 1m I’d expect a normal refresh of the app to re-pull the chart from source, but it doesn’t.

Screenshots

Here’s a snippet of my repo server deployment, for reference on how I’m setting the flag:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
  namespace: argocd
spec:
  template:
    spec:
      containers:
      - command:
        - uid_entrypoint.sh
        - argocd-repo-server
        - --repo-cache-expiration
        - 1m
        - --redis
        - argocd-redis:6379

Version

argocd: v1.5.4+36bade7
  BuildDate: 2020-05-05T19:02:56Z
  GitCommit: 36bade7a2d7b69d1c0b0c4d41191f792a847d61c
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.6.1+159674e
  BuildDate: 2020-06-19T00:41:05Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.14.0

About this issue

Original URL
State: open
Created 4 years ago
Reactions: 12
Comments: 27 (3 by maintainers)

Most upvoted comments

This issue is also related to more cases:

kustomize with remote base (e.g. the manifests of a git url tag)
custom plugin which fetches data from external sources (e.g.: vault secrets: https://github.com/IBM/argocd-vault-plugin)

It would be nice to have a way to cover those use cases without manual intervention (hard-refresh). I think this PR would be helpful here: https://github.com/argoproj/argo-cd/pull/4678

r0bj on Feb 11, 2021

@irishandyb @pierluigilenoci You both bring up valid points, I have a similar dev cycle that re-uses tags, knowing that isn’t the “best practice standard”. All this aside though, I think we need to step back and look at what is actually being asked which would be beneficial in more than one situation, and that is to either: (1) the ability to hard refresh automatically, periodically or much less desirably: (2) the ability to disable cache entirely

The re-use of tags is just one use-case but more generically, being able to re-check the helm chart repository for changes periodically is something that argocd should have the ability to do, like what it currently does with git repositories.

On top of all of this, there’s the point to be made that this is basically a bug, not a feature request, since argocd currently has a cache expiration parameter that straight-up doesn’t work at all. Simply honoring this param would solve these issues. You could set the cache expiration to whatever you want, which, if working, would force argocd to go back to the helm chart repo source to check truth.

De1taE1even on Aug 1, 2022

@irishandyb I understand the particular need but I find it to be a bit overdoing. There are leaner ways to test if a chart is valid and does what it takes than going through ArgoCD. For example by generating the manifests locally or by manually installing them on a cluster.

However, there are tools to automate versioning. For example https://github.com/sstarcher/helm-release And a precise strategy. https://semver.org/#spec-item-9

pierluigilenoci on Aug 1, 2022

@De1taE1even @irishandyb IMHO the discussion here is overlapping different themes:

versioning of the charts
the invalidation of ArgoCD caches
the caching bug that generated this issue

Regarding the versioning of the charts, invalidating the cache is not necessary because Semantic Versioning supports pre-releases and therefore each test can have its own clear tag (and in this way it is also easier to log changes). Let’s say for example that I have the chart version 1.2.3 and I want to make some changes. So I can name every single test version as 1.2.4-alpha-1, 1.2.4-alpha-2 etc (or any combination of dots, alphanumerics, and hyphens). This can also be automated with tools.

As for the cache, having the option to invalidate it with some form of trigger or to disable it entirely is certainly a desirable thing. But it would also be enough just to fix the bug that makes it unusable.

I think it might make sense to separate the feature request to improve cache management from this bug fix request to get things clearer.

pierluigilenoci on Aug 2, 2022

@pierluigilenoci - thank you very much for your input. I will research further and consider changing my process.

That being said, I do agree with @De1taE1even and it would be nice to see some options around caching of Helm charts.

irishandyb on Aug 1, 2022

@irishandyb I am very interested in this problem being solved and obviously everyone is free to use things as they prefer.

IMHO I believe that changing the templates of a chart without changing the version number of the chart itself is against the paradigm of versioning itself. Even if this caching problem gets solved I doubt it solves your problem because solving such anti-paradigm stuff doesn’t make much sense for the community. It basically makes the cache itself useless.

pierluigilenoci on Aug 1, 2022

This issue is also related to more cases:

kustomize with remote base (e.g. the manifests of a git url tag)

custom plugin which fetches data from external sources (e.g.: vault secrets: https://github.com/IBM/argocd-vault-plugin)

It would be nice to have a way to cover those use cases without manual intervention (hard-refresh). I think this PR would be helpful here: #4678

I think we need a way to selectively and periodically hard refresh applications that have some sort of annotation in there.

alwaysastudent on Jul 15, 2021