kyverno: [Bug] Mutating webhook timeout

Kyverno Version

1.6.x

Kubernetes Version

1.23.x

Kubernetes Platform

GKE

Kyverno Rule Type

Mutate

Description

When installing a helm chart, I frequently get the following: Error: Internal error occurred: failed calling webhook "mutate.kyverno.svc-fail": Post "https://kyverno-svc.kyverno.svc:443/mutate?timeout=10s": context deadline exceeded

Steps to reproduce

Policy:

kind: ClusterPolicy
apiVersion: kyverno.io/v1
metadata:
  name: set-daemonset-pod-priority
spec:
  rules:
    - name: set-daemonset-pod-priority
      match:
        any:
        - resources:
            kinds:
            - DaemonSet
      mutate:
        patchStrategicMerge:
          spec:
            template:
              spec:
                +(priorityClassName): "daemonset"

Kyverno values:

replicaCount: 3

resources:
  limits:
    cpu: 400m
    memory: 512Mi
  requests:
    cpu: 400m
    memory: 512Mi

serviceMonitor:
  enabled: true

Expected behavior

Webhook should complete much faster than 10s

Screenshots

No response

Kyverno logs

No response

Slack discussion

No response

Troubleshooting

  • I have read and followed the documentation AND the troubleshooting guide.
  • I have searched other issues in this repository and mine is not recorded.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (7 by maintainers)

Most upvoted comments

We used to have the same issue, but saw client-side throttling up to 12 seconds. We fixed this by setting --clientRateLimitQPS=30 and --clientRateLimitBurst=60. This doesn’t seem to be your problem as far as I can tell from your logs, but wanted to mention it if someone faces the same issue.