argo-cd: Failing to deploy services due to "helm dependency build" failure

Checklist:

• I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I’ve included steps to reproduce the bug.
• I’ve pasted the output of argocd version.

Describe the bug

Some Applications based on helm fail to deploy due to somekind of internal filesystem issue.

For example, one of the apps that are in Unknown states

Name:               redis-rule-hit-count-buffer-test-2
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          test-2
URL:                https://35.232.222.64/applications/redis-rule-hit-count-buffer-test-2
Repo:               git@github.com:gc-org/gc-saas-prod.git
Target:             HEAD
Path:               prod/common/infra/redis
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Unknown
Health Status:      Healthy

CONDITION        MESSAGE                                                                                                                                                                                                                                                  LAST TRANSITION
ComparisonError  rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: unable to move current charts to tmp dir: link error: cannot rename charts to tmpcharts: rename charts tmpcharts: file exists  2020-12-22 10:14:56 +0200 IST

This doesn’t eventually resolve itself, it’s stays this way…

To Reproduce

I’m not sure how to reproduce, this happens from time to time and causes complete deadlock My Chart.yaml

name: redis
version: 0.1.0
apiVersion: v2
dependencies:
  - name: redis
    version: 11.2.2
    repository: https://charts.bitnami.com/bitnami

My app-of-apps

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: redis-servers-test-2
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: git@github.com:gc-org/gc-saas-prod.git
    targetRevision: HEAD
    path: prod/cluster_1/customers/customer_2/redis
    helm:
      releaseName: redis-servers-test-2
  destination:
    server: https://kubernetes.default.svc
    namespace: test-2
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: true

My template

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: redis-dashboard-top-widgets-cache-test-2
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "3"
spec:
  project: default
  source:
    repoURL: git@github.com:gc-org/gc-saas-prod.git
    targetRevision: HEAD
    path: prod/common/infra/redis
    helm:
      values: |
        redis:
          fullnameOverride: redis-dashboard-top-widgets-cache
          redisPort: 6413
          master:
            nodeSelector:
              cus: test-2
            service:redis-rule-hit-count-buffer-master
              port: 6413
          image:
            tag: 6.0.6
          cluster:
            enabled: false
          existingSecret: "redis"
          existingSecretPasswordKey: redis-password
      releaseName: redis-dashboard-top-widgets-cache
  destination:
    server: https://kubernetes.default.svc
    namespace: test-2
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    retry:
      limit: -1 # unlimited
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 5m
    automated:
      prune: true
      selfHeal: true
      allowEmpty: true

Expected behavior

The Application should be deployed successfully

Screenshots

Version

argocd: v1.7.10+bcb05b0.dirty
  BuildDate: 2020-11-21T00:34:29Z
  GitCommit: bcb05b0c2e0f8006aa2d2abaf780e73c9e73c945
  GitTreeState: dirty
  GoVersion: go1.15.5
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.8.1+c2547dc
  BuildDate: 2020-12-10T02:59:21Z
  GitCommit: c2547dca95437fdbb4d1e984b0592e6b9110d37f
  GitTreeState: clean
  GoVersion: go1.14.12
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.8.1 2020-07-16T00:58:46Z
  Helm Version: v3.4.1+gc4e7485
  Kubectl Version: v1.17.8

Logs

Logs from the argocd-application-controller

time="2020-12-22T08:14:58Z" level=info msg="Normalized app spec: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2020-12-22T08:14:56Z\",\"message\":\"rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: unable to move current charts to tmp dir: link error: cannot rename charts to tmpcharts: rename charts tmpcharts: file exists\",\"type\":\"ComparisonError\"}]}}" application=redis-rule-hit-count-buffer-test-2

Logs from argocd-repo-server

time="2020-12-22T08:40:44Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: unable to move current charts to tmp dir: link error: cannot rename charts to tmpcharts: rename charts tmpcharts: file exists" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2020-12-22T08:41:43Z" grpc.service=repository.RepoServerService grpc.start_time="2020-12-22T08:40:44Z" grpc.time_ms=474.433 span.kind=server system=grpc
time="2020-12-22T08:40:44Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:git@github.com:gc-org/gc-saas-prod.git,Path:prod/common/infra/redis,TargetRevision:HEAD,Helm:&ApplicationSourceHelm{ValueFiles:[],Parameters:[]HelmParameter{},ReleaseName:,Values:redis:\n  redisPort: 6380\n  master:\n    service:\n      port: 6380\n  image:\n    tag: 6.0.6\n  nodeSelector:\n    cus: test-1\n  cluster:\n    enabled: false\n  existingSecret: \"redis\"\n  existingSecretPasswordKey: redis-password\n,FileParameters:[]HelmFileParameter{},Version:,},Kustomize:nil,Ksonnet:nil,Directory:nil,Plugin:nil,Chart:,}/cead2aa7818699b6c3ef04fc7e35390ee0fcbee0"