kyverno: [Bug] InternalError, error: Internal error occurred: failed calling webhook "mutate.kyverno.svc-fail"

Kyverno Version

1.8.5

Kubernetes Version

1.23.x

Kubernetes Platform

EKS

Kyverno Rule Type

Mutate

Description

we have eks 1.23 k8s cluster with flux deployment system for implementation gitops approach as part of flux we using kustomizations also we have kyverno installed by helm. after the upgrade from 1.7.5 to 1.8.5 after it we start to ger kustomization errors like this:

NAMESPACE                   NAME                                      AGE    READY     STATUS
app-ns           app-kustomization             5d   False     Deployment/app-ns/app-deployment dry-run failed, reason: InternalError, error: Internal error occurred: failed calling webhook "mutate.kyverno.svc-fail": failed to call webhook: Post "https://kyverno-svc.kyverno.svc:443/mutate?timeout=10s": context deadline exceeded...

as we have kustomization rate is 5 min from time to time it changes to expected behaviour:

app-ns app-kustomization 55d True Applied revision: dev/22122d379a8ed48b742c54f81be9fc2f378b883f

but after it again changes to problem

i checked the kyverno svc and its looks good:

Name:              kyverno-svc
Namespace:         kyverno
Labels:            app=kyverno
Selector:          app.kubernetes.io/instance=kyverno,app.kubernetes.io/name=kyverno,app=kyverno
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                172.20.152.88
IPs:               172.20.152.88
Port:              https  443/TCP
TargetPort:        https/TCP
Endpoints:         100.21.28.93:9443,100.21.67.199:9443,100.21.87.245:9443
Session Affinity:  None
Events:            <none>

we have next version of kustomize-controller - v0.28.0 i have found that if i have 3 kyverno pod replicas and next extraArgs in kyverno deployment configuration:

• –webhookRegistrationTimeout=5m
• –webhookTimeout=30

my kyverno deployment configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "4"
    meta.helm.sh/release-name: kyverno
    meta.helm.sh/release-namespace: kyverno
  labels:
    app: kyverno
    app.kubernetes.io/component: kyverno
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kyverno
    app.kubernetes.io/part-of: kyverno
    app.kubernetes.io/version: v2.6.5
    critical_workload: "true"
    helm.sh/chart: kyverno-v2.6.5
    helm.toolkit.fluxcd.io/name: kyverno
    helm.toolkit.fluxcd.io/namespace: kyverno
  name: kyverno
  namespace: kyverno
spec:
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app.kubernetes.io/instance: kyverno
      app.kubernetes.io/name: kyverno
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 40%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kyverno
        app.kubernetes.io/component: kyverno
        app.kubernetes.io/instance: kyverno
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: kyverno
        app.kubernetes.io/part-of: kyverno
        app.kubernetes.io/version: v2.6.5
        critical_workload: "true"
        helm.sh/chart: kyverno-v2.6.5
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - kyverno
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - args:
        - --autogenInternals=false
        - --backgroundScan=true
        - --webhookRegistrationTimeout=5m
        - --webhookTimeout=30
        env:
        - name: INIT_CONFIG
          value: kyverno
        - name: METRICS_CONFIG
          value: kyverno-metrics
        - name: KYVERNO_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: KYVERNO_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: KYVERNO_SVC
          value: kyverno-svc
        - name: TUF_ROOT
          value: /.sigstore
        - name: KYVERNO_DEPLOYMENT
          value: kyverno
        envFrom:
        - configMapRef:
            name: proxy-environment-variables
        image: 111111111111.dkr.ecr.us-west-2.amazonaws.com/kyverno/kyverno:v1.8.5
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 2
          httpGet:
            path: /health/liveness
            port: 9443
            scheme: HTTPS
          initialDelaySeconds: 15
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 5
        name: kyverno
        ports:
        - containerPort: 9443
          hostPort: 9443
          name: https
          protocol: TCP
        - containerPort: 8000
          hostPort: 8000
          name: metrics-port
          protocol: TCP
        readinessProbe:
          failureThreshold: 6
          httpGet:
            path: /health/readiness
            port: 9443
            scheme: HTTPS
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "3"
            memory: 3G
          requests:
            cpu: "1"
            memory: 2G
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          privileged: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        startupProbe:
          failureThreshold: 12
          httpGet:
            path: /health/liveness
            port: 9443
            scheme: HTTPS
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /.sigstore
          name: sigstore
      dnsPolicy: ClusterFirst
      hostNetwork: true
      initContainers:
      - env:
        - name: METRICS_CONFIG
          value: kyverno-metrics
        - name: KYVERNO_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: KYVERNO_DEPLOYMENT
          value: kyverno
        image: 111111111111.dkr.ecr.us-west-2.amazonaws.com/kyverno/kyvernopre:v1.8.5
        imagePullPolicy: IfNotPresent
        name: kyverno-pre
        resources:
          limits:
            cpu: 100m
            memory: 256Mi
          requests:
            cpu: 10m
            memory: 64Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          privileged: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: kyverno
      serviceAccountName: kyverno
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: monitoring-node
        operator: Equal
        value: "true"
      volumes:
      - emptyDir: {}
        name: sigstore

after the start of kyverno pods it automatically creates MutatingWebhookConfigurations and ValidatingWebhookConfiguration as expected. when i set --autoUpdateWebhooks=false problems with kustomization goes out.

can you suggest how to resolve it? rollback to 1.7.5 didn’t help and problem is still here.

Steps to reproduce

1. upgrade to 1.8.5 from 1.7.5 by using helm
2. start have problems with kustomization
3. downgrade to 1.7.5
4. problems are still here

Expected behavior

successful kustomization all the time.

Screenshots

No response

Kyverno logs

No response

Slack discussion

No response

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.