kyverno: [Bug] Kyverno tries to generate events on blocked resources

Kyverno Version

1.6.x

Kubernetes Version

1.23.x

Kubernetes Platform

K3d

Kyverno Rule Type

Validate

Description

When a validate rule is in enforce mode and blocks a resource which violates the rule, Kyverno tries to generate events on the resource. This results in an unnecessary 10 extra calls to the API server which obviously return 404 errors since they do not exist.

Steps to reproduce

1. Turn logs up to level 6 on the Kyverno Deployment.
2. Create a simple validate rule in enforce mode.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: validate-image-tag
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Using a mutable image tag e.g. 'latest' is not allowed."
      pattern:
        spec:
          containers:
          - image: "!*:latest"

1. Create a simple Pod which violates this.

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: busybox
  name: mypod
spec:
  containers:
  - name: busybox
    image: busybox:latest
    args:
    - "sleep"
    - "9999"

1. See log output below.

Expected behavior

No operations are attempted on a resource which was denied.

Screenshots

No response

Kyverno logs

kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.648027       1 common.go:60] WebhookServer "msg"="spec.ValidationFailureAction set to enforce blocking resource request" "action"="validate" "gvk"="/v1, Kind=Pod" "operation"="CREATE" "resource"="default/Pod/mypod" "policy"="disallow-latest-tag"
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.648099       1 validation.go:105] WebhookServer "msg"="resource blocked" "action"="validate" "gvk"="/v1, Kind=Pod" "operation"="CREATE" "resource"="default/Pod/mypod" 
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.648199       1 server.go:559] WebhookServer/ValidateWebhook "msg"="admission request denied" "kind"="Pod" "name"="mypod" "namespace"="default" "operation"="CREATE" "uid"="4cbcb539-f8a2-4734-8a4c-e16f75cb81ad" 
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.648391       1 server.go:266] WebhookServer/handlerFunc "msg"="admission review request processed" "kind"={"group":"","version":"v1","kind":"Pod"} "name"="mypod" "namespace"="default" "operation"="CREATE" "uid"="4cbcb539-f8a2-4734-8a4c-e16f75cb81ad" "time"="16.95ms"
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.650041       1 event.go:285] Event(v1.ObjectReference{Kind:"ClusterPolicy", Namespace:"", Name:"disallow-latest-tag", UID:"cb0ed6cb-8484-4849-ade9-ac8b3000d742", APIVersion:"kyverno.io/v1", ResourceVersion:"1418389", FieldPath:""}): type: 'Warning' reason: 'PolicyViolation' Rule(s) 'validate-image-tag' failed to apply on resource Pod/default/mypod
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.669455       1 round_trippers.go:553] GET https://10.43.0.1:443/api/v1/namespaces/default/pods/mypod 404 Not Found in 19 milliseconds
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.669982       1 controller.go:140] EventGenerator "msg"="retrying event generation"  "key"={"Kind":"Pod","Name":"mypod","Namespace":"default","Reason":"PolicyViolation","Message":"Rule(s) 'validate-image-tag' of policy 'disallow-latest-tag' failed to apply on the resource","Source":0} "reason"="pods \"mypod\" not found"
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.670770       1 round_trippers.go:553] POST https://10.43.0.1:443/api/v1/namespaces/default/events 201 Created in 21 milliseconds
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.679489       1 round_trippers.go:553] GET https://10.43.0.1:443/api/v1/namespaces/default/pods/mypod 404 Not Found in 6 milliseconds
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.680237       1 controller.go:140] EventGenerator "msg"="retrying event generation"  "key"={"Kind":"Pod","Name":"mypod","Namespace":"default","Reason":"PolicyViolation","Message":"Rule(s) 'validate-image-tag' of policy 'disallow-latest-tag' failed to apply on the resource","Source":0} "reason"="pods \"mypod\" not found"
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.694838       1 round_trippers.go:553] GET https://10.43.0.1:443/api/v1/namespaces/default/pods/mypod 404 Not Found in 11 milliseconds
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.695174       1 controller.go:140] EventGenerator "msg"="retrying event generation"  "key"={"Kind":"Pod","Name":"mypod","Namespace":"default","Reason":"PolicyViolation","Message":"Rule(s) 'validate-image-tag' of policy 'disallow-latest-tag' failed to apply on the resource","Source":0} "reason"="pods \"mypod\" not found"
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.705169       1 round_trippers.go:553] GET https://10.43.0.1:443/api/v1/namespaces/default/pods/mypod 404 Not Found in 5 milliseconds
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.705452       1 controller.go:140] EventGenerator "msg"="retrying event generation"  "key"={"Kind":"Pod","Name":"mypod","Namespace":"default","Reason":"PolicyViolation","Message":"Rule(s) 'validate-image-tag' of policy 'disallow-latest-tag' failed to apply on the resource","Source":0} "reason"="pods \"mypod\" not found"
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.720199       1 round_trippers.go:553] GET https://10.43.0.1:443/api/v1/namespaces/default/pods/mypod 404 Not Found in 4 milliseconds
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.720506       1 controller.go:140] EventGenerator "msg"="retrying event generation"  "key"={"Kind":"Pod","Name":"mypod","Namespace":"default","Reason":"PolicyViolation","Message":"Rule(s) 'validate-image-tag' of policy 'disallow-latest-tag' failed to apply on the resource","Source":0} "reason"="pods \"mypod\" not found"
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.741807       1 round_trippers.go:553] GET https://10.43.0.1:443/api/v1/namespaces/default/pods/mypod 404 Not Found in 4 milliseconds
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.742102       1 controller.go:140] EventGenerator "msg"="retrying event generation"  "key"={"Kind":"Pod","Name":"mypod","Namespace":"default","Reason":"PolicyViolation","Message":"Rule(s) 'validate-image-tag' of policy 'disallow-latest-tag' failed to apply on the resource","Source":0} "reason"="pods \"mypod\" not found"
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.779822       1 round_trippers.go:553] GET https://10.43.0.1:443/api/v1/namespaces/default/pods/mypod 404 Not Found in 4 milliseconds
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.780073       1 controller.go:140] EventGenerator "msg"="retrying event generation"  "key"={"Kind":"Pod","Name":"mypod","Namespace":"default","Reason":"PolicyViolation","Message":"Rule(s) 'validate-image-tag' of policy 'disallow-latest-tag' failed to apply on the resource","Source":0} "reason"="pods \"mypod\" not found"
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.849818       1 round_trippers.go:553] GET https://10.43.0.1:443/api/v1/namespaces/default/pods/mypod 404 Not Found in 4 milliseconds
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.850065       1 controller.go:140] EventGenerator "msg"="retrying event generation"  "key"={"Kind":"Pod","Name":"mypod","Namespace":"default","Reason":"PolicyViolation","Message":"Rule(s) 'validate-image-tag' of policy 'disallow-latest-tag' failed to apply on the resource","Source":0} "reason"="pods \"mypod\" not found"
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.982955       1 round_trippers.go:553] GET https://10.43.0.1:443/api/v1/namespaces/default/pods/mypod 404 Not Found in 4 milliseconds
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:57.983225       1 controller.go:140] EventGenerator "msg"="retrying event generation"  "key"={"Kind":"Pod","Name":"mypod","Namespace":"default","Reason":"PolicyViolation","Message":"Rule(s) 'validate-image-tag' of policy 'disallow-latest-tag' failed to apply on the resource","Source":0} "reason"="pods \"mypod\" not found"
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:58.244359       1 round_trippers.go:553] GET https://10.43.0.1:443/api/v1/namespaces/default/pods/mypod 404 Not Found in 3 milliseconds
kyverno-5ccddb664f-w6l2w kyverno I0222 12:39:58.244615       1 controller.go:140] EventGenerator "msg"="retrying event generation"  "key"={"Kind":"Pod","Name":"mypod","Namespace":"default","Reason":"PolicyViolation","Message":"Rule(s) 'validate-image-tag' of policy 'disallow-latest-tag' failed to apply on the resource","Source":0} "reason"="pods \"mypod\" not found"

Slack discussion

No response

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.