kyverno: [Bug] Webhook API version hard-coded to v1beta1

Kyverno Version

1.8.0

Kubernetes Version

1.25.x

Kubernetes Platform

Other (specify in description)

Kyverno Rule Type

Mutate

Description

Cluster running rke2 (v1.25.2+rke2r1).

The following policy makes the mutation webhook deny the creation of any pods since upgrading from K8S 1.24 to 1.25:

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: apply-pod-label-pod-name
  annotations:
    policies.kyverno.io/title: Apply pod-name Labels to Pod
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      This policy creates pod-name label on selected Pods.
spec:
  mutateExistingOnPolicyUpdate: true
  generateExistingOnPolicyUpdate: true
  failurePolicy: Ignore
  rules:
    - name: pod
      match:
        any:
          - resources:
              kinds:
                - Pod
      exclude:
        any:
          - resources:
              selector:
                matchLabels:
                  job-name: "*"
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              snapshot.cluster.arpa/pod-name: "{{ request.object.metadata.name }}"

The short version of the error is "msg"="mutation failed" "error"="mutation policy apply-pod-label-pod-name error: failed to validate resource mutated by policy apply-pod-label-pod-name: pre-validation: couldn't find model , err: invalid kind" "kind"="Pod" "name"="" "namespace"="productivity" "operation"="CREATE" "uid"="f7231912-5939-4cec-82b6-202ee580d99e" (full logs below).

It appears that the pod name is empty, which should not be the case; this makes the webhook fail to apply the policy and block the deployment of the pod.

After investigation, it looks like the AdmissionReviewVersions parameter of the kyverno-resource-mutating-webhook-cfg is hardcoded to v1beta1, and thus doesn’t support well v1:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  creationTimestamp: "2022-10-13T21:55:11Z"
  generation: 7
  labels:
    webhook.kyverno.io/managed-by: kyverno
  name: kyverno-resource-mutating-webhook-cfg
  ownerReferences:
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    name: kyverno:webhook
    uid: b324595b-6965-4cb8-b1f3-cc6047ae086f
  resourceVersion: "82523799"
  uid: dd7b0d4d-e64e-4eae-8cdc-39a58330f244
webhooks:
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    caBundle: REMOVED
    service:
      name: kyverno-svc
      namespace: kyverno
      path: /mutate/ignore
      port: 443
  failurePolicy: Ignore
  matchPolicy: Equivalent
  name: mutate.kyverno.svc-ignore
  namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values:
      - kyverno
  objectSelector: {}
  reinvocationPolicy: IfNeeded
  rules:
  - apiGroups:
    - '*'
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - '*/*'
    scope: '*'
  sideEffects: NoneOnDryRun
  timeoutSeconds: 10

Adding the --autoUpdateWebhooks=false flag to the kyverno deployment, and manually adding v1to the list of admissionReviewVersions resolves the issue.

I believe the following would need to be updated to support v1 as well:

https://github.com/kyverno/kyverno/blob/b7247b5935d789827eaabc85544928fade595653/pkg/controllers/webhook/controller.go#L494 https://github.com/kyverno/kyverno/blob/b7247b5935d789827eaabc85544928fade595653/pkg/controllers/webhook/controller.go#L521 https://github.com/kyverno/kyverno/blob/b7247b5935d789827eaabc85544928fade595653/pkg/controllers/webhook/controller.go#L542 https://github.com/kyverno/kyverno/blob/b7247b5935d789827eaabc85544928fade595653/pkg/controllers/webhook/controller.go#L567 https://github.com/kyverno/kyverno/blob/b7247b5935d789827eaabc85544928fade595653/pkg/controllers/webhook/controller.go#L621 https://github.com/kyverno/kyverno/blob/b7247b5935d789827eaabc85544928fade595653/pkg/controllers/webhook/controller.go#L640 https://github.com/kyverno/kyverno/blob/b7247b5935d789827eaabc85544928fade595653/pkg/controllers/webhook/controller.go#L675 https://github.com/kyverno/kyverno/blob/b7247b5935d789827eaabc85544928fade595653/pkg/controllers/webhook/controller.go#L728 https://github.com/kyverno/kyverno/blob/b7247b5935d789827eaabc85544928fade595653/pkg/controllers/webhook/controller.go#L746

References:

Steps to reproduce

  1. Create the ClusterPolicyfrom the description above on a cluster running K8S v1.25.x (bug identified on v1.25.2+rke2r1 specifically, not sure if other k8s platforms are impacted).
  2. Deploy a new pod which matches the policy
  3. The new pod will not be able to deploy as blocked by the mutating webhook

Expected behavior

The policy should be correctly applied and in this example, add the correct label to the pod.

Screenshots

No response

Kyverno logs

###### Using `AdmissionReviewVersions: ["v1beta1"]` in `MutatingWebhookConfiguration` ######

kyverno-5658c55474-7vcqp kyverno I1013 22:14:39.409549       1 handlers.go:159] webhooks/resource/mutate "msg"="received an admission request in mutating webhook" "kind"="Pod" "name"="" "namespace"="productivity" "operation"="CREATE" "uid"="f7231912-5939-4cec-82b6-202ee580d99e"
kyverno-5658c55474-7vcqp kyverno I1013 22:14:39.409744       1 handlers.go:166] webhooks/resource/mutate "msg"="processing policies for mutate admission request" "kind"="Pod" "mutatePolicies"=1 "name"="" "namespace"="productivity" "operation"="CREATE" "uid"="f7231912-5939-4cec-82b6-202ee580d99e" "verifyImagesPolicies"=0
kyverno-5658c55474-7vcqp kyverno I1013 22:14:39.414051       1 logr.go:261] context "msg"="Adding service account" "service account name"="replicaset-controller" "service account namespace"="kube-system"
kyverno-5658c55474-7vcqp kyverno I1013 22:14:39.414313       1 logr.go:261]  "msg"="updated image info" "images"={"containers":{"redmine":{"registry":"ghcr.io","name":"redmine","path":"fastlorenzo/redmine","digest":"sha256:40be52c02b34bea3ce58da3ad515c811f638b9b308fb05a82075d78cf92a7eba"}},"initContainers":{"init-db":{"registry":"ghcr.io","name":"postgres-initdb","path":"onedr0p/postgres-initdb","tag":"14.5"}}}
kyverno-5658c55474-7vcqp kyverno I1013 22:14:39.416653       1 logr.go:261]  "msg"="applied JSON patch" "patch"=[{"op":"replace","path":"/spec/initContainers/0/image","value":"ghcr.io/onedr0p/postgres-initdb:14.5"},{"op":"replace","path":"/spec/containers/0/image","value":"ghcr.io/fastlorenzo/redmine@sha256:40be52c02b34bea3ce58da3ad515c811f638b9b308fb05a82075d78cf92a7eba"}]
kyverno-5658c55474-7vcqp kyverno I1013 22:14:39.427125       1 mutation.go:107] webhooks/resource/mutate "msg"="applying policy mutate rules" "kind"="Pod" "name"="" "namespace"="productivity" "operation"="CREATE" "policy"="apply-pod-label-pod-name" "uid"="f7231912-5939-4cec-82b6-202ee580d99e"
kyverno-5658c55474-7vcqp kyverno I1013 22:14:39.427227       1 mutation.go:33] EngineMutate "msg"="start mutate policy processing" "kind"="Pod" "name"="" "namespace"="productivity" "policy"="apply-pod-label-pod-name" "startTime"="2022-10-13T22:14:39.42720248Z"
kyverno-5658c55474-7vcqp kyverno I1013 22:14:39.427387       1 mutation.go:61] EngineMutate "msg"="processing mutate rule" "applyRules"="All" "kind"="Pod" "name"="" "namespace"="productivity" "policy"="apply-pod-label-pod-name" "rule"="pod"
kyverno-5658c55474-7vcqp kyverno I1013 22:14:39.436730       1 mutation.go:108] EngineMutate "msg"="apply rule to resource" "kind"="Pod" "name"="" "namespace"="productivity" "policy"="apply-pod-label-pod-name" "resource name"="" "resource namespace"="productivity" "rule"="pod"
kyverno-5658c55474-7vcqp kyverno I1013 22:14:39.440735       1 mutation.go:319] EngineMutate "msg"="finished processing policy" "kind"="Pod" "mutationRulesApplied"=0 "name"="" "namespace"="productivity" "policy"="apply-pod-label-pod-name" "processingTime"="13.516494ms"
kyverno-5658c55474-7vcqp kyverno E1013 22:14:39.440785       1 handlers.go:180] webhooks/resource/mutate "msg"="mutation failed" "error"="mutation policy apply-pod-label-pod-name error: failed to validate resource mutated by policy apply-pod-label-pod-name: pre-validation: couldn't find model , err: invalid kind" "kind"="Pod" "name"="" "namespace"="productivity" "operation"="CREATE" "uid"="f7231912-5939-4cec-82b6-202ee580d99e"

###### Using `AdmissionReviewVersions: ["v1beta1","v1"]` in `MutatingWebhookConfiguration` ######

kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.659108       1 handlers.go:159] webhooks/resource/mutate "msg"="received an admission request in mutating webhook" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "operation"="UPDATE" "uid"="4830b513-367e-406d-9cb5-0eb5aa4e474f"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.659145       1 handlers.go:166] webhooks/resource/mutate "msg"="processing policies for mutate admission request" "kind"="Pod" "mutatePolicies"=1 "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "operation"="UPDATE" "uid"="4830b513-367e-406d-9cb5-0eb5aa4e474f" "verifyImagesPolicies"=0
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.666631       1 logr.go:261] context "msg"="Adding service account" "service account name"="canal" "service account namespace"="kube-system"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.667037       1 logr.go:261]  "msg"="updated image info" "images"={"containers":{"redmine":{"registry":"ghcr.io","name":"redmine","path":"fastlorenzo/redmine","digest":"sha256:40be52c02b34bea3ce58da3ad515c811f638b9b308fb05a82075d78cf92a7eba"}},"initContainers":{"init-db":{"registry":"ghcr.io","name":"postgres-initdb","path":"onedr0p/postgres-initdb","tag":"14.5"}}}
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.669898       1 logr.go:261]  "msg"="applied JSON patch" "patch"=[{"op":"replace","path":"/spec/containers/0/image","value":"ghcr.io/fastlorenzo/redmine@sha256:40be52c02b34bea3ce58da3ad515c811f638b9b308fb05a82075d78cf92a7eba"},{"op":"replace","path":"/spec/initContainers/0/image","value":"ghcr.io/onedr0p/postgres-initdb:14.5"}]
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.679870       1 mutation.go:107] webhooks/resource/mutate "msg"="applying policy mutate rules" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "operation"="UPDATE" "policy"="apply-pod-label-pod-name" "uid"="4830b513-367e-406d-9cb5-0eb5aa4e474f"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.679937       1 mutation.go:33] EngineMutate "msg"="start mutate policy processing" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "policy"="apply-pod-label-pod-name" "startTime"="2022-10-13T22:47:08.679920145Z"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.680051       1 mutation.go:61] EngineMutate "msg"="processing mutate rule" "applyRules"="All" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "policy"="apply-pod-label-pod-name" "rule"="pod"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.690848       1 mutation.go:108] EngineMutate "msg"="apply rule to resource" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "policy"="apply-pod-label-pod-name" "resource name"="redmine-76d9bc5976-ds2lk" "resource namespace"="productivity" "rule"="pod"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.700322       1 vars.go:380] EngineMutate "msg"="variable substituted" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "path"="/mutate/patchStrategicMerge/metadata/labels/snapshot.cluster.arpa/pod-name" "policy"="apply-pod-label-pod-name" "rule"="pod" "value"="redmine-76d9bc5976-ds2lk" "variable"="{{ request.object.metadata.name }}"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.700427       1 strategicMergePatch.go:21] EngineMutate/ProcessStrategicMergePatch "msg"="started applying strategicMerge patch" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "policy"="apply-pod-label-pod-name" "rule"="pod" "startTime"="2022-10-13T22:47:08.700406607Z"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.701484       1 strategicMergePatch.go:100] EngineMutate/ProcessStrategicMergePatch "msg"="applying strategic merge patch" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "patch"="{\"metadata\": {\"labels\": {\"snapshot.cluster.arpa/pod-name\": \"redmine-76d9bc5976-ds2lk\"}}}\n" "policy"="apply-pod-label-pod-name" "rule"="pod"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.706904       1 strategicMergePatch.go:63] EngineMutate "msg"="generating JSON patches from patched resource" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "patchedResource"={"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"cni.projectcalico.org/containerID":"08cb93883430abfee7b62ffeefdd31d903d99005c5b43634d58cb4f2c43f9f40","cni.projectcalico.org/podIP":"10.42.1.188/32","cni.projectcalico.org/podIPs":"10.42.1.188/32"},"creationTimestamp":"2022-10-13T22:47:00Z","generateName":"redmine-76d9bc5976-","labels":{"app.kubernetes.io/component":"redmine","app.kubernetes.io/instance":"redmine","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"redmine","helm.sh/chart":"redmine-20.3.7","pod-template-hash":"76d9bc5976","snapshot.cluster.arpa/pod-name":"redmine-76d9bc5976-ds2lk"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1",... REMOVED_FOR_BREVITY...}} "policy"="apply-pod-label-pod-name" "rule"="pod"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.707529       1 strategicMergePatch.go:75] EngineMutate "msg"="generated patch" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "patch"="{\"op\":\"add\",\"path\":\"/metadata/labels/snapshot.cluster.arpa~1pod-name\",\"value\":\"redmine-76d9bc5976-ds2lk\"}" "policy"="apply-pod-label-pod-name" "rule"="pod"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.707548       1 strategicMergePatch.go:28] EngineMutate/ProcessStrategicMergePatch "msg"="finished applying strategicMerge patch" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "policy"="apply-pod-label-pod-name" "processingTime"="7.131068ms" "rule"="pod"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.717347       1 mutation.go:319] EngineMutate "msg"="finished processing policy" "kind"="Pod" "mutationRulesApplied"=1 "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "policy"="apply-pod-label-pod-name" "processingTime"="37.401806ms"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.717652       1 mutation.go:118] webhooks/resource/mutate "msg"="mutation rules from policy applied successfully" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "operation"="UPDATE" "policy"="apply-pod-label-pod-name" "rules"=["pod"] "uid"="4830b513-367e-406d-9cb5-0eb5aa4e474f"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.717707       1 annotations.go:135] webhooks/resource/mutate "msg"="annotation value prepared" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "operation"="UPDATE" "patches"=[{"rulename":"pod","op":"add","path":"/metadata/labels/snapshot.cluster.arpa~1pod-name"}] "uid"="4830b513-367e-406d-9cb5-0eb5aa4e474f"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.717766       1 mutation.go:171] webhooks/resource/mutate "msg"="created patches" "count"=2 "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "operation"="UPDATE" "uid"="4830b513-367e-406d-9cb5-0eb5aa4e474f"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.717786       1 mutation.go:74] webhooks/resource/mutate "msg"="" "generated patches"="[{\"op\":\"add\",\"path\":\"/metadata/labels/snapshot.cluster.arpa~1pod-name\",\"value\":\"redmine-76d9bc5976-ds2lk\"}, {\"path\":\"/metadata/annotations/policies.kyverno.io~1last-applied-patches\",\"op\":\"add\",\"value\":\"pod.apply-pod-label-pod-name.kyverno.io: added /metadata/labels/snapshot.cluster.arpa~1pod-name\\n\"}]" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "operation"="UPDATE" "uid"="4830b513-367e-406d-9cb5-0eb5aa4e474f"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.719120       1 utils.go:40] webhooks/resource/mutate "msg"="" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "operation"="UPDATE" "patchedResource"="{\"kind\":\"Pod\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"redmine-76d9bc5976-ds2lk\",\"generateName\":\"redmine-76d9bc5976-\",\"namespace\":\"productivity\",\"uid\":\"c3f5447d-9967-4d00-aa56-9fbb17a43284\",\"resourceVersion\":\"82535549\",\"creationTimestamp\":\"2022-10-13T22:47:00Z\",\"labels\":{\"app.kubernetes.io/component\":\"redmine\",\"app.kubernetes.io/instance\":\"redmine\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"redmine\",\"helm.sh/chart\":\"redmine-20.3.7\",\"pod-template-hash\":\"76d9bc5976\",\"snapshot.cluster.arpa/pod-name\":\"redmine-76d9bc5976-ds2lk\"},\"annotations\":{\"cni.projectcalico.org/containerID\":\"08cb93883430abfee7b62ffeefdd31d903d99005c5b43634d58cb4f2c43f9f40\",\"cni.projectcalico.org/podIP\":\"10.42.1.188/32\",\"cni.projectcalico.org/podIPs\":\"10.42.1.188/32\",\"policies.kyverno.io/last-applied-patches\":\"pod.apply-pod-label-pod-name.kyverno.io: added /metadata/labels/snapshot.cluster.arpa~1pod-name\\n\"},\"ownerReferences\":[{\"apiVersion\":\... REMOVED_FOR_BREVITY...}}" "uid"="4830b513-367e-406d-9cb5-0eb5aa4e474f"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.719182       1 handler.go:51] webhooks/resource/mutate "msg"="images verified" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "operation"="UPDATE" "patches"="" "uid"="4830b513-367e-406d-9cb5-0eb5aa4e474f" "warnings"=null
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.719219       1 handlers.go:197] webhooks/resource/mutate "msg"="completed mutating webhook" "kind"="Pod" "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "operation"="UPDATE" "response"={"uid":"","allowed":true,"patch":"W3sib3AiOiJhZGQiLCJwYXRoIjoiL21ldGFkYXRhL2xhYmVscy9zbmFwc2hvdC5jbHVzdGVyLmFycGF+MXBvZC1uYW1lIiwidmFsdWUiOiJyZWRtaW5lLTc2ZDliYzU5NzYtZHMybGsifSwgeyJwYXRoIjoiL21ldGFkYXRhL2Fubm90YXRpb25zL3BvbGljaWVzLmt5dmVybm8uaW9+MWxhc3QtYXBwbGllZC1wYXRjaGVzIiwib3AiOiJhZGQiLCJ2YWx1ZSI6InBvZC5hcHBseS1wb2QtbGFiZWwtcG9kLW5hbWUua3l2ZXJuby5pbzogYWRkZWQgL21ldGFkYXRhL2xhYmVscy9zbmFwc2hvdC5jbHVzdGVyLmFycGF+MXBvZC1uYW1lXG4ifV0="} "uid"="4830b513-367e-406d-9cb5-0eb5aa4e474f"
kyverno-6784bf4f99-qs5t9 kyverno I1013 22:47:08.719641       1 admission.go:88] webhooks/resource/mutate "msg"="admission review request processed" "kind"={"group":"","version":"v1","kind":"Pod"} "name"="redmine-76d9bc5976-ds2lk" "namespace"="productivity" "operation"="UPDATE" "time"="61.145155ms" "uid"="4830b513-367e-406d-9cb5-0eb5aa4e474f"

Slack discussion

No response

Troubleshooting

  • I have read and followed the documentation AND the troubleshooting guide.
  • I have searched other issues in this repository and mine is not recorded.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

It looks a bit risky at first sight but if it works 👍 Closing, please reopen or create a new issue if needed.

+1
eddycharly on Oct 14, 2022

FWIW we were planning to switch to v1 in 1.9. It’s not as simple as changing the version, we need to do it gracefully when upgrading.

+1
eddycharly on Oct 14, 2022
Read more comments on GitHub
← kyverno: [Bug] verifyImages wrongly claims image is not signed
kyverno: [Bug] Webhook Controller Endless Loop →