kyverno: [Bug] Kyverno creates too many admissionReports

Kyverno Version

1.9.0

Description

Kyverno creates too many AdmissionReports resulting into etcd running out of space, and cluster becoming unresponsive.

This might somehow be related to one of the clusterpolicies on our clusters matching on *

More details in slack thread.

Current workaround: Setting admissionReports=false

Slack discussion

https://kubernetes.slack.com/archives/CLGR9BJU9/p1677785359616429

Troubleshooting

I have read and followed the documentation AND the troubleshooting guide.
I have searched other issues in this repository and mine is not recorded.

About this issue

Original URL
State: closed
Created a year ago
Comments: 39 (25 by maintainers)

Most upvoted comments

backgroundScanWorkers is for background scan, it does not influence admission reports controller.

eddycharly on Mar 13, 2023

backgroundScanWorkers does not control the number of workers in the admission reports controller.

1.9.2-rc.1 is being cut, please give it a try.

eddycharly on Mar 13, 2023

Issue seems to be related to running Kyverno on large clusters (or rules that match on *) with low or default QPS/Burst settings. Closing this for now, as it doesn’t seem to be a bug, but rather a misconfiguration.

UPDATE: Figured I should also update this and note that one can monitor/alert on etcd_db_total_size_in_bytes to potentially prevent this in advance

L1ghtman2k on Mar 7, 2023