cert-manager: CA-injector doc updates( was Webhook patching infinite loop)

Describe the bug:

I’m using the cainjector to patch a mutating webhook.

The config for the webhook:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-apiserver-ca: "true"
    cert-manager.io/inject-ca-from: istio-system/istiod
    ...
    generation: 1823361
    resourceVersion: "3014242" (keeps changing)

I have an Issuer in istio-system:

apiVersion: v1
items:
- apiVersion: cert-manager.io/v1alpha2
  kind: Issuer
  metadata:
    annotations:
      cert-manager.io/certificate-name: istio-ca
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"cert-manager.io/v1alpha2","kind":"Issuer","metadata":{"annotations":{"cert-manager.io/certificate-name":"istio-ca"},"name":"istio-issuer","namespace":"istio-system"},"spec":{"ca":{"secretName":"istio-certmanager-ca"}}}
    creationTimestamp: "2020-05-05T17:41:40Z"
    generation: 1
    name: istio-issuer
    namespace: istio-system
    resourceVersion: "1008062"
    selfLink: /apis/cert-manager.io/v1alpha2/namespaces/istio-system/issuers/istio-issuer
    uid: 4fe3440c-d0fc-4335-927e-186cebaef38d
  spec:
    ca:
      secretName: istio-certmanager-ca
  status:
    conditions:
    - lastTransitionTime: "2020-05-05T17:41:40Z"
      message: Signing CA verified
      reason: KeyPairVerified
      status: "True"
      type: Ready
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

And a certificate:

apiVersion: v1
items:
- apiVersion: cert-manager.io/v1alpha2
  kind: Certificate
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{},"name":"istiod","namespace":"istio-system"},"spec":{"commonName":"istiod.istio-system.svc","dnsNames":["istiod.istio-system.svc"],"issuerRef":{"kind":"Issuer","name":"istio-issuer"},"secretName":"istiod"}}
    creationTimestamp: "2020-05-05T20:50:36Z"
    generation: 1
    name: istiod
    namespace: istio-system
    resourceVersion: "1015054"
    selfLink: /apis/cert-manager.io/v1alpha2/namespaces/istio-system/certificates/istiod
    uid: 43d201fa-e83f-42f6-9463-0bd7aa56e80e
  spec:
    commonName: istiod.istio-system.svc
    dnsNames:
    - istiod.istio-system.svc
    issuerRef:
      kind: Issuer
      name: istio-issuer
    secretName: istiod
  status:
    conditions:
    - lastTransitionTime: "2020-05-05T20:50:36Z"
      message: Certificate is up to date and has not expired
      reason: Ready
      status: "True"
      type: Ready
    notAfter: "2020-08-03T20:50:23Z"

Expected behaviour: CA patched into the webhookconfig - once.

Steps to reproduce the bug: Steps to reproduce the bug should be clear and easily reproducible to help people gain an understanding of the problem.

Anything else we need to know?: Logs:

I0507 21:06:59.064027       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="test-webhook" "resource_namespace"="" 
I0507 21:06:59.064090       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"test-webhook"}
I0507 21:06:59.064186       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="test-webhook" "resource_namespace"="" 
I0507 21:06:59.064236       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"test-webhook"}
I0507 21:06:59.064702       1 sources.go:94] cert-manager/inject-controller "msg"="Extracting CA from Certificate resource" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="test-webhook" "resource_namespace"="" "certificate"="istio-system/istiod"
I0507 21:06:59.074842       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="test-webhook" "resource_namespace"="" 
I0507 21:06:59.074874       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"test-webhook"}
I0507 21:06:59.078005       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="test-webhook" "resource_namespace"="" 
I0507 21:06:59.078044       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"test-webhook"}
I0507 21:06:59.078108       1 sources.go:94] cert-manager/inject-controller "msg"="Extracting CA from Certificate resource" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="test-webhook" "resource_namespace"="" "certificate"="istio-system/istiod"
I0507 21:06:59.087592       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="test-webhook" "resource_namespace"="" 
I0507 21:06:59.087638       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"test-webhook"}
I0507 21:06:59.088602       1 sources.go:94] cert-manager/inject-controller "msg"="Extracting CA from Certificate resource" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="test-webhook" "resource_namespace"="" "certificate"="istio-system/istiod"
I0507 21:06:59.088784       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="test-webhook" "resource_namespace"="" 
I0507 21:06:59.088819       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"test-webhook"}

Environment details::

Kubernetes version (e.g. v1.10.2): 1.17.4
Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): local (using kind and rancher)
cert-manager version (e.g. v0.4.0): quay.io/jetstack/cert-manager-cainjector:v0.15.0
Install method (e.g. helm or static manifests): static manifests

/kind bug

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 16 (2 by maintainers)

Most upvoted comments

I will reopen this bug since I think the code should have at least some backoff/detection. It’s low priority - but the injector (as singleton) could have a map with the configs that it already patched as key and a timestamp, and if it detects the same config is patched again backoff and warn.

Even if the doc is fixed - and some validation is added to not allow 2 conflicting patching - it is possible that other external components may attempt to patch the config.

I would also suggest some code to verify that caBundle is the same - and skip update otherwise. It seems possible to run the caInjector in different namespaces and with different options - including with disabled lead election. This will also result in non-stop updates, even if they apply the same change.

costinm on May 12, 2020