argo-cd: argo-repo-server issue: gpg ... --gen-key failed exit status 2

I managed to get everything running. But I did a complete fresh setup using the install.yaml from v2.4.13. There were quite some changes so it’s not easy to say which diff was the important one.

We had the same issue (argocd-repo-server erroring out with GPG errors) when installing from kustomize from cluster-install manifest. The setup works off the shelf for minikube, but for a “real” cluster had this issue. Just removing the seccompProfile section from the securityContext section solved it.

          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault

We experienced the same error after upgrading from 2.2.x to 2.4.11.

In our case we had patched the deployment with the below patch. After removing it, the error disappeared and repo server could start up.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault

I’m using the “official” argo-cd Helm chart to deploy ArgoCD on a K8s cluster. Unfortunately I cannot “unset” the seccompProfile value because of a bug in Helm when trying to overring subchart values (https://github.com/helm/helm/issues/5184, https://github.com/helm/helm/issues/9136, current pull request fixing this problem:https://github.com/helm/helm/pull/11440). Normally, you just have to set the seccompProfile to “null”. This happen because I created a custom Helm chart which has a dependency the official ArgoCD helm chart.

So, for me the only solution was to set a value (at least this works when setting values for subcharts). If someone encounters the same problem, just set the following value in your values.yaml file of your umbrella/parent chart:

argo-cd :   <= correspong to the name of my Helm "umbrella" chart, might be different for your
  repoServer:
    containerSecurityContext:
      seccompProfile:
        type: Unconfined

Hopping a better solution can be found in the near future because having to lower security to make the service work is not really a great workaround !

We @swisscom have the same issue with a Kubernetes cluster based on VMware Tanzu v1.21.9+vmware.1.

  Kernel Version:             4.15.0-167-generic
  OS Image:                   Ubuntu 16.04.7 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.9
  Kubelet Version:            v1.21.9+vmware.1
  Kube-Proxy Version:         v1.21.9+vmware.1

Whoa.

Log Line

time="2022-10-06T09:10:58Z" level=info msg="gpg --no-permission-warning --logger-fd 1 --batch --gen-key /tmp/gpg-key-recipe3419147315" dir= execID=0ca0f
time="2022-10-06T09:11:04Z" level=debug msg="gpg: can't connect to the agent: End of file\ngpg: agent_genkey failed: No agent running\ngpg: key generation failed: No agent running\n" duration=6.01130389s execID=0ca0f
time="2022-10-06T09:11:04Z" level=error msg="`gpg --no-permission-warning --logger-fd 1 --batch --gen-key /tmp/gpg-key-recipe3419147315` failed exit status 2" execID=0ca0f
time="2022-10-06T09:11:04Z" level=info msg=Trace args="[gpg --no-permission-warning --logger-fd 1 --batch --gen-key /tmp/gpg-key-recipe3419147315]" dir= operation_name="exec gpg" time_ms=6011.9106090000005
time="2022-10-06T09:11:04Z" level=fatal msg="`gpg --no-permission-warning --logger-fd 1 --batch --gen-key /tmp/gpg-key-recipe3419147315` failed exit status 2"

argocd-repo-server

v2.4.12

gpg

argocd@argocd-repo-server-64d5df97c5-2p6xx:~$ gpg --version
gpg (GnuPG) 2.2.27
libgcrypt 1.9.4
Copyright (C) 2021 Free Software Foundation, Inc.
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /home/argocd/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

argocd@argocd-repo-server-64d5df97c5-2p6xx:~$ ldd $(which gpg)
	linux-vdso.so.1 (0x00007ffe69139000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1a538c1000)
	libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f1a538ae000)
	libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f1a53761000)
	libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f1a53623000)
	libreadline.so.8 => /lib/x86_64-linux-gnu/libreadline.so.8 (0x00007f1a535cf000)
	libassuan.so.0 => /lib/x86_64-linux-gnu/libassuan.so.0 (0x00007f1a535b9000)
	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f1a53591000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1a53369000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1a53282000)
	libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f1a53250000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1a539e4000)

Hm, also no change for us. So far only disabling gpg as described in https://argo-cd.readthedocs.io/en/stable/user-guide/gpg-verification/ removes the error.

We experienced the same error after upgrading from 2.2.x to 2.4.11.

In our case we had patched the deployment with the below patch. After removing it, the error disappeared and repo server could start up.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault

I have the same problem when I upgraded ArgoCD to version 2.5.2 This workaround works for me. Thanks bro!

I managed to get everything running. But I did a complete fresh setup using the install.yaml from v2.4.13. There were quite some changes so it’s not easy to say which diff was the important one.

Hi brother, I also met the same problem. The final solution is to check whether your argocd version and kubernetes version correspond. Refer to the link https://argo-cd.readthedocs.io/en/stable/operator-manual/installation/

We had 2.4.11 running in a temporary playground environment where it ran fine. Then we started comparing the setups and found the seccompProfile was missing in the playground environment. We removed it in our production environments and Bingo!

We @swisspost are also facing this issue on VMware TKGI 1.22 (= TKGI 1.13.4-build.15). Argo CD v2.4.7 (via helm chart version 4.9.16) is working fine on AWS EKS and Azure AKS but not on TKGI.

TKGI runs really old worker OS:

$ kubectl get nodes -o wide
NAME     STATUS   ROLES    AGE     VERSION            INTERNAL-IP    EXTERNAL-IP    OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
08f...   Ready    <none>   4h20m   v1.22.6+vmware.1   172.23.129.8   172.23.129.8   Ubuntu 16.04.7 LTS   4.15.0-176-generic   docker://20.10.9

I then suspected that Ubuntu 16 worker with Ubuntu 22 base image has some compat issues (Argo 2.4 container image is based on Ubuntu 22).

Unfortunately this theory is wrong - I booted a single-node k3s cluster with Ubuntu 16 LTS, docker and k3s:

$ kubectl get nodes -o wide
NAME     STATUS   ROLES                  AGE   VERSION        INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
ubuntu   Ready    control-plane,master   16m   v1.22.6+k3s1   192.168.91.205   <none>        Ubuntu 16.04.7 LTS   4.4.0-186-generic   docker://20.10.7

$ kubectl -n argocd get po
NAME                                  READY   STATUS    RESTARTS   AGE
argocd-redis-6bb9c5d89f-kh4jj         1/1     Running   0          5m57s
argocd-application-controller-0       1/1     Running   0          5m57s
argocd-repo-server-7d97f5cbdb-5tqjc   1/1     Running   0          5m56s
argocd-server-9f646bf78-qfsf4         1/1     Running   0          5m56s

$ helm ls -n argocd
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
argocd  argocd          1               2022-09-23 17:02:28.2194322 +0200 CEST  deployed        argo-cd-4.9.16  v2.4.7

Argo CD 2.4.7 is working fine here. I have no clue what else to try and filed a VMware issue in our support portal. 🙄

I can confirm it’s definitely missing special rights for this service account. I granted now everything for this service account and now the argocd-repo-server is starting. I’m now digging into that we have a least privilege role.

@denysvitali Could you please try out in your env by excluding the seccomProfile ?

We experienced the same error after upgrading from 2.2.x to 2.4.11.

In our case we had patched the deployment with the below patch. After removing it, the error disappeared and repo server could start up.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault

just removing the seccompProfile sub-section worked for us (details). Thank you, @jabbors ! How did you narrow down to this part?

We @swisscom have the same issue with a Kubernetes cluster based on VMware Tanzu v1.21.9+vmware.1.

  Kernel Version:             4.15.0-167-generic
  OS Image:                   Ubuntu 16.04.7 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.9
  Kubelet Version:            v1.21.9+vmware.1
  Kube-Proxy Version:         v1.21.9+vmware.1

Whoa.

Nice! Thanks for confirming that❤️. We postponed the upgrade intent for Argo on Tanzu and wait for TKGI 1.16 in Q1 2023 🙄