kyverno: [Bug] Mutating webhook timeout
Kyverno Version
1.6.x
Kubernetes Version
1.23.x
Kubernetes Platform
GKE
Kyverno Rule Type
Mutate
Description
When installing a helm chart, I frequently get the following:
Error: Internal error occurred: failed calling webhook "mutate.kyverno.svc-fail": Post "https://kyverno-svc.kyverno.svc:443/mutate?timeout=10s": context deadline exceeded
Steps to reproduce
Policy:
kind: ClusterPolicy
apiVersion: kyverno.io/v1
metadata:
name: set-daemonset-pod-priority
spec:
rules:
- name: set-daemonset-pod-priority
match:
any:
- resources:
kinds:
- DaemonSet
mutate:
patchStrategicMerge:
spec:
template:
spec:
+(priorityClassName): "daemonset"
Kyverno values:
replicaCount: 3
resources:
limits:
cpu: 400m
memory: 512Mi
requests:
cpu: 400m
memory: 512Mi
serviceMonitor:
enabled: true
Expected behavior
Webhook should complete much faster than 10s
Screenshots
No response
Kyverno logs
No response
Slack discussion
No response
Troubleshooting
- I have read and followed the documentation AND the troubleshooting guide.
- I have searched other issues in this repository and mine is not recorded.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (7 by maintainers)
We used to have the same issue, but saw client-side throttling up to 12 seconds. We fixed this by setting
--clientRateLimitQPS=30
and--clientRateLimitBurst=60
. This doesn’t seem to be your problem as far as I can tell from your logs, but wanted to mention it if someone faces the same issue.