argo-cd: ApplicationSet controller receives x509: certificate signed by unknown authority, when using SCM, PR or Git generator with private bitbucket server

Slack conversation thread : https://cloud-native.slack.com/archives/C01U45M2SVB/p1654569552211179 Checklist:

I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
I’ve included steps to reproduce the bug.
I’ve pasted the output of argocd version.

Describe the bug Receive x509: certificate signed by unknown authority when application set controller with an SCM generator attempts to connect to a private bitbucket server.

To Reproduce

Follow installation for release v2.4.0-rc4 under https://github.com/argoproj/argo-cd/releases
Follow steps to setup TLS certificates as documented https://argo-cd.readthedocs.io/en/release-1.8/user-guide/private-repositories/#self-signed-untrusted-tls-certificates
Create an ApplicationSet CRD with an SCM generator for the Bitbucket Server SCM. The bitbucket server needs to be a private repository with TLS connectivity via self signed certificates which should have been registered in the above step

Expected behavior

Expected the application set controller to be able to communicate with bitbucket server API’s to list repositories and automate ArgoCD application generation.

Screenshots

Version

{
    "Version": "v2.4.0-rc2+cd5b2af",
    "BuildDate": "2022-05-18T12:42:45Z",
    "GitCommit": "cd5b2af358f3574ea151fr8c24b1d52dafe59c85",
    "GitTreeState": "clean",
    "GoVersion": "go1.18.2",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v4.4.1 2021-11-11T23:36:27Z",
    "HelmVersion": "v3.8.1+g5cb9af4",
    "KubectlVersion": "v0.23.1",
    "JsonnetVersion": "v0.18.0"
}

Logs

applicationset-controller logs

time="2022-06-08T00:45:28Z" level=info msg="Starting configmap/secret informers"
time="2022-06-08T00:45:28Z" level=info msg="Configmap/secret informer synced"
time="2022-06-08T00:45:28Z" level=info msg="Starting webhook server"
time="2022-06-08T00:45:28Z" level=info msg="Starting manager"
time="2022-06-08T00:45:29Z" level=error msg="error generating params" error="error listing repos: error listing repositories for: Get \"https://<myBitBucketServerHost>/rest/api/1.0/projects/<myProject>/repos?limit=100\": x509: certificate signed by unknown authority" generator="&{0xc000c6c1e0 <nil>}"

Volume and Volume Mounts of the applicationset-controller

 Mounts:
      /app/config/ssh from ssh-known-hosts (rw)
      /app/config/tls from tls-certs (rw)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from argocd-applicationset-controller-token-59cjk (ro)


Volumes:
  tls-certs:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      argocd-tls-certs-cm
    Optional:  false
  ssh-known-hosts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      argocd-ssh-known-hosts-cm
    Optional:  false

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 3
Comments: 21 (3 by maintainers)

Most upvoted comments

Yeah it’s a bug

rishabh625 on Jun 8, 2022

Ran into this issue today, this is the workaround I found (keep in mind we’re using an on premise setup of argocd and gitlab and thus don’t need the usual certificates):

In the values.yaml of the argocd deployment add:

applicationSet: extraVolumeMounts: - mountPath: /etc/ssl/certs name: tls-certs

This will mount the tls certificates from argocd’s config-map into the default ssl certificate folder of golang’s http client, getting around the issue. It does however override the other certificates so this isn’t ideal, maybe someone can further improve on it in the future, at least until an option is made available to append these tls certificates.

Best of luck! Guy

guydolinksy on Jan 5, 2023

+1, this is a showstopper for us. We’re experiencing this issue when using a Helm chart source with a cert signed by our custom CA, despite having explicitly mounted the cert chain in /etc/ssl/certs on the filesystem of the pods of the controller, server, repo server and applicationset controller. Argo CD 2.5.3+0c7de21 installed with helm chart v5.9.1

hnikt-jonasfh on Jan 4, 2023

a blocker for us to use applicationsets. is there a way to allow unsafe certificates instead of trusting them?

tabnul on Dec 21, 2022

We have the same issue under Openshift 4.13 running the Openshift GitOps Operator (ArgoCD v2.6.7). I connected repos from both BitBucket and GitLab on-prem servers with HTTPS and successfully created applicationsets with the “git”-generator. When doing the same with the “scmProvider” generator, I get the message:

  conditions:
  - lastTransitionTime: "2023-08-08T11:31:09Z"
    message: 'error listing repos: error listing repositories for REPO: Get "https://<url>/rest/api/1.0/projects/REPO/repos?limit=100":
      x509: certificate signed by unknown authority'
    reason: ApplicationGenerationFromParamsError
    status: "True"
    type: ErrorOccurred
  - lastTransitionTime: "2023-08-08T11:31:09Z"
    message: 'error listing repos: error listing repositories for REPO: Get "https://<url>/rest/api/1.0/projects/REPO/repos?limit=100":
      x509: certificate signed by unknown authority'
    reason: ErrorOccurred
    status: "False"
    type: ParametersGenerated
  - lastTransitionTime: "2023-08-08T11:31:09Z"
    message: 'error listing repos: error listing repositories for REPO: Get "https://<url>/rest/api/1.0/projects/REPO/repos?limit=100":
      x509: certificate signed by unknown authority'
    reason: ApplicationGenerationFromParamsError
    status: "False"
    type: ResourcesUpToDate

The repositories inside ArgoCD are not “insecure”, so the certificates are implemented correct. But the scmProvider generator obviously has problems with them.

ThomasVitt on Aug 8, 2023