cert-manager: Endless Sync Loop when installing Helm Chart via ArgoCD

Describe the bug: Helm chart’s ValidatingWebhookConfiguration is missing a configuration section which Kubernetes adds in on Azure Kubernetes Service to every ValidatingWebhookConfiguration which causes ArgoCD to place the Helm chart into a constant sync loop and never become healthy.

Expected behaviour: This ValidatingWebhookConfiguration

Steps to reproduce the bug: Install this Application CRD on your cluster running ArgoCD 2.0.3 on an Azure Kubernetes Service cluster:

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  name: cert-manager
spec:
  project: default
  destination:
    namespace: cert-manager
    server: 'https://kubernetes.default.svc'
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  source:
    chart: cert-manager
    repoURL: 'https://charts.jetstack.io'
    targetRevision: v1.4.0
    helm:
      releaseName: cert-manager
      values: |+
        global:
          logLevel: 2
        ingressShim:
          defaultIssuerName: letsencrypt
          defaultIssuerKind: ClusterIssuer
        installCRDs: true
        prometheus:
          enabled: true
          servicemonitor:
            enabled: true

Once you do, this section will repeatedly want to delete webhooks[0].namespaceSelector.matchExpressions[2]:

- key: control-plane
  operator: DoesNotExist

Anything else we need to know?: Nope.

Environment details::

Kubernetes version: 1.20.7
Cloud-provider/provisioner: Azure Kubernetes Service
cert-manager version: v1.4.0
Install method: e.g. helm/static manifests: ArgoCD 2.0.3 using above Application CRD instance.

/kind bug

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 5
Comments: 22 (4 by maintainers)

Commits related to this issue

https://github.com/cert-manager/cert-manager/issues/4114 — committed to joaquinrz/capi-gitops by deleted user a year ago
merge alessandro dev branch (#4) * argocd self-manage * infra name * policy * root-on root * name * argocd app and values * exclude values * exclude * multi sources * root ... — committed to joaquinrz/capi-gitops by ams0 a year ago
Disable addmission enforcer, see https://github.com/cert-manager/cert-manager/issues/4114 — committed to joachimprinzbach/aks-infra-argocd-cluster-apps by joachimprinzbach a year ago

Most upvoted comments

I worked around this using ArgoCD IgnoreDifferences. This way you won’t have to change the cert-manager resources. See the ArgoCD Docs for more information

ignoreDifferences:
  - group: admissionregistration.k8s.io
    kind: ValidatingWebhookConfiguration
    name: cert-manager-webhook
    jqPathExpressions:
      - .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "control-plane")

edit: On 23 September 2023 it looks like Microsoft Azure (AKS) added an extra matchExpressions item with the key kubernetes.azure.com/managedby. We changed above ignoreDifferences to the following, which resolved the issue for us:

ignoreDifferences:
  - group: admissionregistration.k8s.io
    kind: ValidatingWebhookConfiguration
    name: cert-manager-webhook
    jqPathExpressions:
      - .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "control-plane")
      - .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "kubernetes.azure.com/managedby")

+24

hijmen on Sep 25, 2023

Had the same problem after adding it to argocd using kustomize. In my case, I added the following patch:

patchesJson6902:
- target:
    group: admissionregistration.k8s.io
    version: v1
    kind: ValidatingWebhookConfiguration
    name: cert-manager-webhook
  patch: |-
    - op: add
      path: /webhooks/0/namespaceSelector/matchExpressions/-
      value:
        key: control-plane
        operator: DoesNotExist

jemag on Oct 29, 2021

I found another alternative.

In their documentation AKS & microsoft explain you can disable the Admissions Enforcer that is adding that matchExpression with either a label or an annotation.

I simply dropped this into my helm configuration

webhook:
  validatingWebhookConfigurationAnnotations:
    admissions.enforcer/disabled: "true"

And the diff is gone. I know AKS doesn’t recommend you doing this since it can affect protected namespaces but given this webhook is configured to only affect cert-manager CRD’s and not vanilla k8s resources this is a better solution for people like me who uses ApplicationAets and changing one single app is not feasible.

tete17 on Jun 2, 2022

@wallrj
requesting for a review for the PR: https://github.com/cert-manager/website/pull/1350

shahkv95 on Nov 25, 2023

/reopen

We should document the work around for this in:

https://cert-manager.io/docs/installation/continuous-deployment-and-gitops/

xref: cert-manager/website#320

Hi @wallrj
I would like to take this issue and update the existing document with the workaround for cd with argocd. Could you pls assign the issue to me?

Thanks

shahkv95 on Nov 24, 2023

It’d be nice if https://github.com/cert-manager/cert-manager/issues/4114#issuecomment-1008162907 was documented in the installation instructions somewhere. – I’d consider that a "fix’ for this.

Alternatively, perhaps we can just get this fixed in ArgoCD: https://github.com/argoproj/argo-cd/issues/4276#issuecomment-907797060

jsoref on Feb 3, 2022