kyverno: [Bug] 1.10 backgroundscanreports are not created

Kyverno Version

1.10.0

Description

We’ve used v1.9 previously and now trying v1.10 in our test cluster. This is a clean installation with only admissionController and reportsController. I.e values.yaml like this:

cleanupJobs:
  admissionReports:
    enabled: false
  clusterAdmissionReports:
    enabled: false

backgroundController:
  enabled: false

cleanupController:
  enabled: false

reportsController:
  metricsService:
    create: false

admissionController:
  metricsService:
    create: false
  metering:
    disabled: true  

We then deploy some of our policies (Enforce) which some Pods in the cluster should violate. But background reports are never created:

$ k get kyverno -A
NAMESPACE   NAME                                                      BACKGROUND   VALIDATE ACTION   READY   AGE   MESSAGE
            clusterpolicy.kyverno.io/check-priority-class             true         Enforce           True    12h   Ready
...
            
NAMESPACE   NAME                                                              AGE   PASS   FAIL   WARN   ERROR   SKIP
it          admissionreport.kyverno.io/1fc7a8e6-e3e2-485a-9040-181caa48cb36   10m   8      1      0      0       7

$ k get backgroundscanreport -A
No resources found

We’ve tried to wait for 1h, re-create policy etc, check all the namespace exceptions, --skipResourceFilters=false etc. In logs:

$ k logs -f --tail=10 kyverno-reports-controller-56cf8b5f5f-z6dnk
E0531 08:00:45.191419       1 discovery.go:270] dynamic-client "msg"="failed to find preferred resource version" "error"=null
E0531 08:00:45.262558       1 discovery.go:270] dynamic-client "msg"="failed to find preferred resource version" "error"=null
E0531 08:00:45.460184       1 discovery.go:270] dynamic-client "msg"="failed to find preferred resource version" "error"=null
E0531 08:00:45.526954       1 discovery.go:270] dynamic-client "msg"="failed to find preferred resource version" "error"=null
E0531 08:00:45.723396       1 discovery.go:270] dynamic-client "msg"="failed to find preferred resource version" "error"=null
E0531 08:00:45.796479       1 discovery.go:270] dynamic-client "msg"="failed to find preferred resource version" "error"=null
E0531 08:00:45.974703       1 discovery.go:270] dynamic-client "msg"="failed to find preferred resource version" "error"=null
E0531 08:00:46.022771       1 discovery.go:270] dynamic-client "msg"="failed to find preferred resource version" "error"=null
E0531 08:01:13.745084       1 discovery.go:270] dynamic-client "msg"="failed to find preferred resource version" "error"=null
E0531 08:18:33.199732       1 discovery.go:270] dynamic-client "msg"="failed to find preferred resource version" "error"=null
^C

If I enable debug logs (-v=9) in reports controller:

...
2023-05-31 10:23:13 | I0531 08:23:13.340972       1 round_trippers.go:553] GET https://10.88.148.1:443/api/namespaces/kube-system/hubble-ui-55d7978649 404 Not Found in 1 milliseconds |  
2023-05-31 10:23:13 | I0531 08:23:13.339399       1 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json" -H "User-Agent: reports-controller/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer <masked>" 'https://10.88.148.1:443/api/namespaces/kube-system/hubble-ui-55d7978649' |  
2023-05-31 10:23:13 | E0531 08:23:13.339251       1 discovery.go:270] dynamic-client "msg"="failed to find preferred resource version" "error"=null
...

all these 404 errors are seems to be related to existing ReplicaSets or pods of DaemonSet. But these errors I cannot understand what is the resource is?

2023-05-31 11:10:27 | background-scan-controller/worker "msg"="Dropping request from the queue" "error"="the server could not find the requested resource" "id"=1 "obj"="kyverno/da7c0d73-1446-4741-8b06-fcba7e9a3922" |  
2023-05-31 11:10:27 | background-scan-controller/worker "msg"="done" "duration"="195.004713ms" "id"=1 "key"="kyverno/da7c0d73-1446-4741-8b06-fcba7e9a3922" "name"="da7c0d73-1446-4741-8b06-fcba7e9a3922" "namespace"="kyverno" |  
2023-05-31 11:10:27 | dynamic-client "msg"="failed to find preferred resource version" "error"=null

v1.9 with the same policies in this cluster was able to generate reports in every namespace. How can we find the reason of not creating reports?

Slack discussion

No response

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.