argo-cd: ignoreDifferences in namespaceSelector in Webhook

Hello, I am trying to use argoCD to deploy seldon-core-operator

seldon-core-operator installs a mutatingwebhookconfiguration and a validatingwebhookconfiguration

After installation, the live manifest has an additional namespace selector as shown below

The path to the namespaceSelector as validated in the live mutatingwebhookconfiguration using yq is

$ cat liveMutatingWebhookConfiguration.yaml | yq r - "webhooks[0].namespaceSelector.matchExpressions[1]"
key: control-plane
operator: DoesNotExist

So I tried adding the following ignoreDifferences spec

- group: admissionregistration.k8s.io
    kind: MutatingWebhookConfiguration
    name: seldon-mutating-webhook-configuration-{{ .Values.global.spec.destination.namespace }}
    namespace: {{ .Values.global.spec.destination.namespace }}
    jsonPointers:
    - /webhooks/0/clientConfig/caBundle
    - /webhooks/0/namespaceSelector/matchExpressions/1

However, this does not seem to resolve the difference and argoCD still reports the namespaceSelector as different and shows the webhookconfigurations as OutOfSync

How can I fix this?

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 11
Comments: 21 (2 by maintainers)

Most upvoted comments

I had the same problem with cert-manager on AKS, and tweaked @fdebuire 's suggestion to use jqPathExpressions, which also worked:

 ignoreDifferences:
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      name: cert-manager-webhook
      jqPathExpressions:
        - .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "control-plane")

+11

jschnurr on Aug 29, 2021

I used:

    - kind: ValidatingWebhookConfiguration
      group: admissionregistration.k8s.io
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle
        - /webhooks/1/clientConfig/caBundle
        - /webhooks/2/clientConfig/caBundle
        - /webhooks/3/clientConfig/caBundle
        - /webhooks/4/clientConfig/caBundle
        - /webhooks/5/clientConfig/caBundle
        - /webhooks/6/clientConfig/caBundle
        - /webhooks/7/clientConfig/caBundle

+10

vortarian on Dec 10, 2020

If you find your way here while trying to use ArgoCD to configure Istio, and are getting an OutOfSync error on istiod-default-validator, the following configuration in an ArgoCD app that installs the istio/base Helm chart will help:

spec:
  ...
  ignoreDifferences:     
  - group: admissionregistration.k8s.io                                                              
    kind: ValidatingWebhookConfiguration
    jsonPointers:                                                                                    
    - /webhooks/0/clientConfig/caBundle
    - /webhooks/0/failurePolicy

SafeEval on Oct 16, 2022

I’m on Argo v2.1.1 deployed with kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.1.1/manifests/install.yaml, and the following manifest works for me using the argoproj.io/v1alpha1 apiVersion.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: cert-manager
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://charts.jetstack.io
    chart: cert-manager
    targetRevision: 1.5.3
    helm:
      values: |
        installCRDs: true
        ingressShim:
          defaultIssuerName: letsencrypt
          defaultIssuerKind: ClusterIssuer
        nodeSelector:
          kubernetes.io/os: linux
  # https://github.com/argoproj/argo-cd/issues/4276
  ignoreDifferences:
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      name: cert-manager-webhook
      jqPathExpressions:
        - .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "control-plane")
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

jschnurr on Sep 6, 2021

I used:

    - kind: ValidatingWebhookConfiguration
      group: admissionregistration.k8s.io
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle
        - /webhooks/1/clientConfig/caBundle
        - /webhooks/2/clientConfig/caBundle
        - /webhooks/3/clientConfig/caBundle
        - /webhooks/4/clientConfig/caBundle
        - /webhooks/5/clientConfig/caBundle
        - /webhooks/6/clientConfig/caBundle
        - /webhooks/7/clientConfig/caBundle

Seems like could be solved with one-liner here:

- kind: ValidatingWebhookConfiguration
  group: admissionregistration.k8s.io
  jqPathExpressions:
    - '.webhooks[]?.clientConfig.caBundle'

punkerpunker on Jun 21, 2022

I trying to apply the above example on argocd-2.1.2 and I am getting the following error.

error: error validating "argocd/application.yaml": error validating data: [ValidationError(Application.spec.ignoreDifferences[0]): unknown field "jqPathExpressions" in io.argoproj.v1alpha1.Application.spec.ignoreDifferences, ValidationError(Application.spec.ignoreDifferences[0]): missing required field "jsonPointers" in io.argoproj.v1alpha1.Application.spec.ignoreDifferences]; if you choose to ignore these errors, turn validation off with --validate=false

Kubernetes-1.20.0

rdgacarvalho on Sep 14, 2021

Got into the same issue with cert-manager on AKS (https://github.com/Azure/AKS/issues/1771) and this is what worked for me:

ignoreDifferences:
  - group: admissionregistration.k8s.io
    kind: ValidatingWebhookConfiguration
    name: cert-manager-webhook
    jsonPointers:
      - /webhooks/0/namespaceSelector/matchExpressions/2

However when I tried to narrow to the specific resource by adding the optional namespace filter it doesn’t work anymore.

fdebuire on Jun 4, 2021

I ran into this issue again playing with Argo v2.9 (v2.9.0-rc2) that started failing on this solution due to an "invalid patch."I couldn’t find a format that would make ArgoCD happy, but ultimately moving it to the declarative config map (https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/) did the trick. For reference:

# apply a similar configmap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.ignoreDifferences.admissionregistration.k8s.io_ValidatingWebhookConfiguration: |
    jsonPointers:
    - /webhooks/0/clientConfig/caBundle
    - /webhooks/0/failurePolicy
    jqPathExpressions:
    - .webhooks[0].clientConfig.caBundle
    - .webhooks[0].failurePolicy

# restart argocd
kubectl -n argocd rollout restart deployment argocd-applicationset-controller argocd-notifications-controller argocd-redis argocd-repo-server argocd-server

Inspiration: https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#system-level-configuration

jimangel on Oct 12, 2023

In my case I have an app of apps deployment in ArgoCD.

The System Level Diffing configuration that is documented in the ArgoCD docs worked fine for the parent application that is not out-of-sync anymore. But unfortunately this is not the case for the application that the MutatingWebhookConfiguration belongs to.

Does this mean that I have to put this ignoreDifferences on every application that has the Mutating/ValidatingWebhookConfiguration ?

marcgascon on Dec 10, 2020