kyverno: Kyverno's endpoints are not available after kyverno chart installation succeeds and as a result kyverno-policies chart installation fails and cluster installation fails

Kyverno Version

1.9.2

Kubernetes Version

1.24.x

Kubernetes Platform

Other (specify in description)

Kyverno Rule Type

Validate

Description

kyverno version v1.9.3 (chart version: 2.7.3) Kubernetes Platform: https://github.com/gardener/gardener

Issue (I think this is the issue, but you know better.) kyverno-resource-validating-webhook-cfg is created after more then 5 minutes after the kyverno Pod created after kyverno installed.

Impact on the system: Cluster installation fails, because kyverno-policies chart installation fails. Error: conditions not met: condition Ready is in status False: chart release api-gw-center-ugw-kyverno-policies is in failed status: retry 1: could not install chart: release api-gw-center-ugw-kyverno-policies failed, and has been uninstalled due to atomic being set: Internal error occurred: failed calling webhook “validate-policy.kyverno.svc”: failed to call webhook: Post “https://api-gw-center-kyverno-svc.kyverno.svc:443/policyvalidate?timeout=10s”: context deadline exceeded

Steps to reproduce

1. install kyverno chart
2. install kyverno-policies chart (immediately after the step 1)

Expected behavior

expected: both kyverno and kyverno-policies charts are installed successfully

actual: kyverno and kyverno-policies charts are installed, but kyverno-policies chart is installed with error and it breaks our cluster installation process

Screenshots

Kyverno logs

2023-05-24T18:57:31.098848071Z {"level":"info","ts":1684954651.098696,"logger":"setup.version","caller":"version/version.go:17","msg":"version","version":"v1.9.3","hash":"/dab311d6b3d75ce0b20e6166dabc479b08d5c98e","time":"2023-05-09_01:51:14PM"}
2023-05-24T18:57:31.098887866Z {"level":"info","ts":1684954651.098752,"logger":"setup.maxprocs","caller":"internal/maxprocs.go:12","msg":"setup maxprocs..."}
2023-05-24T18:57:31.099389843Z {"level":"info","ts":1684954651.099309,"logger":"setup.maxprocs","caller":"internal/maxprocs.go:16","msg":"maxprocs: Updating GOMAXPROCS=1: determined from CPU quota"}
2023-05-24T18:57:31.099471843Z {"level":"info","ts":1684954651.0994115,"logger":"setup.kube-client","caller":"internal/client.go:37","msg":"create kube client...","kubeconfig":"","qps":300,"burst":300}
2023-05-24T18:57:31.099923213Z {"level":"info","ts":1684954651.0998447,"logger":"setup.signals","caller":"internal/signal.go:16","msg":"setup signals..."}
2023-05-24T18:57:31.100049551Z {"level":"info","ts":1684954651.0999904,"logger":"setup.metrics","caller":"internal/metrics.go:25","msg":"setup metrics...","otel":"prometheus","port":"8000","collector":"opentelemetrycollector.kyverno.svc.cluster.local","creds":""}
2023-05-24T18:57:31.100058060Z {"level":"info","ts":1684954651.1000097,"logger":"setup.metrics","caller":"internal/metrics.go:17","msg":"load metrics configuration..."}
2023-05-24T18:57:31.115181065Z {"level":"info","ts":1684954651.1150851,"logger":"setup.kube-client","caller":"internal/client.go:37","msg":"create kube client...","kubeconfig":"","qps":300,"burst":300}
2023-05-24T18:57:31.115818948Z {"level":"info","ts":1684954651.11574,"logger":"setup.kube-client","caller":"internal/client.go:37","msg":"create kube client...","kubeconfig":"","qps":300,"burst":300}
2023-05-24T18:57:31.116378865Z {"level":"info","ts":1684954651.1162963,"logger":"setup.kyverno-client","caller":"internal/client.go:45","msg":"create kyverno client...","kubeconfig":"","qps":300,"burst":300}
2023-05-24T18:57:31.116659911Z {"level":"info","ts":1684954651.1165993,"logger":"setup.metadata-client","caller":"internal/client.go:61","msg":"create metadata client...","kubeconfig":"","qps":300,"burst":300}
2023-05-24T18:57:31.116898766Z {"level":"info","ts":1684954651.1168478,"logger":"setup.dynamic-client","caller":"internal/client.go:53","msg":"create dynamic client...","kubeconfig":"","qps":300,"burst":300}
2023-05-24T18:57:31.417341962Z {"level":"info","ts":1684954651.4172018,"logger":"setup.registry-client","caller":"kyverno/main.go:74","msg":"setup registry client...","secrets":"","insecure":false}
2023-05-24T18:57:31.417367413Z {"level":"info","ts":1684954651.4172657,"logger":"setup.cosign","caller":"kyverno/main.go:90","msg":"setup cosign...","repository":""}
2023-05-24T18:57:32.183856115Z {"level":"info","ts":1684954652.1837158,"logger":"EventGenerator","caller":"event/controller.go:119","msg":"start"}
2023-05-24T18:57:32.183897747Z {"level":"info","ts":1684954652.1837683,"logger":"setup.controllers","caller":"internal/controller.go:32","msg":"starting controller","name":"policycache-controller","workers":3}
2023-05-24T18:57:32.183984559Z {"level":"info","ts":1684954652.1838872,"logger":"setup.controllers","caller":"internal/controller.go:32","msg":"starting controller","name":"openapi-controller","workers":1}
2023-05-24T18:57:32.184292400Z {"level":"info","ts":1684954652.1841986,"logger":"setup.controllers","caller":"internal/controller.go:32","msg":"starting controller","name":"config-controller","workers":3}
2023-05-24T18:57:32.184525192Z {"level":"info","ts":1684954652.1844544,"logger":"klog","caller":"leaderelection/leaderelection.go:248","msg":"attempting to acquire leader lease kyverno/kyverno...\n"}
2023-05-24T18:57:32.197944113Z {"level":"info","ts":1684954652.1978376,"logger":"klog","caller":"leaderelection/leaderelection.go:258","msg":"successfully acquired lease kyverno/kyverno\n"}
2023-05-24T18:57:32.201188540Z {"level":"info","ts":1684954652.2010942,"logger":"setup.leader-election","caller":"leaderelection/leaderelection.go:97","msg":"still leading","id":"api-gw-center-kyverno-78d5cfd469-svl45"}
2023-05-24T18:57:32.201314769Z {"level":"info","ts":1684954652.2012475,"logger":"setup.leader-election","caller":"leaderelection/leaderelection.go:83","msg":"started leading","id":"api-gw-center-kyverno-78d5cfd469-svl45"}
2023-05-24T18:57:32.284809111Z {"level":"info","ts":1684954652.284643,"logger":"webhooks","caller":"webhooks/server.go:195","msg":"starting service"}
2023-05-24T18:57:32.424052891Z {"level":"info","ts":1684954652.4238791,"logger":"setup.leader.controllers","caller":"internal/controller.go:32","msg":"starting controller","name":"background-scan-controller","workers":2}
2023-05-24T18:57:32.424176768Z {"level":"info","ts":1684954652.4240625,"logger":"setup.leader.controllers","caller":"internal/controller.go:32","msg":"starting controller","name":"policy-controller","workers":2}
2023-05-24T18:57:32.424198611Z {"level":"info","ts":1684954652.4240918,"logger":"PolicyController","caller":"policy/policy_controller.go:291","msg":"starting"}
2023-05-24T18:57:32.424207613Z {"level":"info","ts":1684954652.4241285,"logger":"klog","caller":"cache/shared_informer.go:273","msg":"Waiting for caches to sync for PolicyController\n"}
2023-05-24T18:57:32.424212523Z {"level":"info","ts":1684954652.4241667,"logger":"klog","caller":"cache/shared_informer.go:280","msg":"Caches are synced for PolicyController\n"}
2023-05-24T18:57:32.424345189Z {"level":"info","ts":1684954652.4242575,"logger":"setup.leader.controllers","caller":"internal/controller.go:32","msg":"starting controller","name":"certmanager-controller","workers":1}
2023-05-24T18:57:32.424469451Z {"level":"info","ts":1684954652.4243965,"logger":"setup.leader.controllers","caller":"internal/controller.go:32","msg":"starting controller","name":"webhook-controller","workers":2}
2023-05-24T18:57:32.424765459Z {"level":"info","ts":1684954652.4245214,"logger":"setup.leader.controllers","caller":"internal/controller.go:32","msg":"starting controller","name":"exception-webhook-controller","workers":1}
2023-05-24T18:57:32.425461726Z {"level":"info","ts":1684954652.425309,"logger":"setup.leader.controllers","caller":"internal/controller.go:32","msg":"starting controller","name":"background-controller","workers":10}
2023-05-24T18:57:32.425473463Z {"level":"info","ts":1684954652.425348,"logger":"background","caller":"background/update_request_controller.go:119","msg":"starting"}
2023-05-24T18:57:32.425477336Z {"level":"info","ts":1684954652.4253995,"logger":"klog","caller":"cache/shared_informer.go:273","msg":"Waiting for caches to sync for background\n"}
2023-05-24T18:57:32.425502261Z {"level":"info","ts":1684954652.425425,"logger":"klog","caller":"cache/shared_informer.go:280","msg":"Caches are synced for background\n"}
2023-05-24T18:57:32.425556641Z {"level":"info","ts":1684954652.425498,"logger":"setup.leader.controllers","caller":"internal/controller.go:32","msg":"starting controller","name":"resource-report-controller","workers":1}
2023-05-24T18:57:32.425694332Z {"level":"info","ts":1684954652.4255843,"logger":"setup.leader.controllers","caller":"internal/controller.go:32","msg":"starting controller","name":"aggregate-report-controller","workers":1}
2023-05-24T18:57:32.425737477Z {"level":"info","ts":1684954652.425688,"logger":"setup.leader.controllers","caller":"internal/controller.go:32","msg":"starting controller","name":"admission-report-controller","workers":10}
2023-05-24T19:00:54.407807234Z {"level":"info","ts":1684954854.4066203,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"21e60288-50cc-4e21-8ac7-379ccaae5768","kind":"ClusterPolicy","name":"disallow-host-path"}
2023-05-24T19:00:54.416373567Z {"level":"info","ts":1684954854.416259,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"a69bd17d-c2df-4f50-83ee-670bf51fb7c6","kind":"ClusterPolicy","name":"require-run-as-non-root"}
2023-05-24T19:00:54.441726391Z {"level":"info","ts":1684954854.4416053,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"3f074b8b-a2a2-42ae-a930-315073bf1289","kind":"ClusterPolicy","name":"disallow-container-sock-mounts"}
2023-05-24T19:00:54.458178895Z {"level":"info","ts":1684954854.4578843,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"9050d88b-f6d0-4f75-aa1c-2087dc1aee3f","kind":"ClusterPolicy","name":"require-requests-limits"}
2023-05-24T19:00:54.495028148Z {"level":"info","ts":1684954854.4947114,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"03923286-dc47-48d1-8bb2-17b2c8d3c834","kind":"ClusterPolicy","name":"restrict-apparmor-profiles"}
2023-05-24T19:00:54.522933688Z {"level":"info","ts":1684954854.522804,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"89846686-a876-4849-aaaf-645d60d72a9d","kind":"ClusterPolicy","name":"disallow-host-ports"}
2023-05-24T19:00:54.530233142Z {"level":"info","ts":1684954854.5299032,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"6307da90-a166-4b63-82b6-48392467c543","kind":"ClusterPolicy","name":"disallow-selinux"}
2023-05-24T19:00:54.580085879Z {"level":"info","ts":1684954854.5799434,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"578c6ab6-5034-4784-932f-67f40e7b6b62","kind":"ClusterPolicy","name":"restrict-seccomp"}
2023-05-24T19:00:54.591534283Z {"level":"info","ts":1684954854.5914137,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"5f33fe71-bb5f-48f0-a740-bf9074be86eb","kind":"ClusterPolicy","name":"disallow-host-namespaces"}
2023-05-24T19:00:54.628911529Z {"level":"info","ts":1684954854.626939,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"c7862a52-6c0b-4e55-adee-fa61bb45ddca","kind":"ClusterPolicy","name":"deny-privilege-escalation"}
2023-05-24T19:00:54.640445637Z {"level":"info","ts":1684954854.6403341,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"aaa5cdeb-b104-4b8f-afbd-c18294cefeec","kind":"ClusterPolicy","name":"no-localhost-service"}
2023-05-24T19:00:54.692795713Z {"level":"info","ts":1684954854.692652,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"fe6faf79-c090-4575-8c0a-5d5b1c39dbf3","kind":"ClusterPolicy","name":"disallow-add-capabilities"}
2023-05-24T19:00:54.742405214Z {"level":"info","ts":1684954854.7422328,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"e9c3de6d-56c7-443a-b280-3eff4ce947c1","kind":"ClusterPolicy","name":"restrict-cert-client-properties"}
2023-05-24T19:00:54.742429810Z {"level":"info","ts":1684954854.7423184,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"c0bcb338-7560-4c01-a27c-fe4fc7a0dbc7","kind":"ClusterPolicy","name":"require-image-checksum"}
2023-05-24T19:00:54.774784539Z {"level":"info","ts":1684954854.774657,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"ad518042-a192-4664-9b62-c93f486658f8","kind":"ClusterPolicy","name":"disallow-default-namespace"}
2023-05-24T19:00:54.782550956Z {"level":"info","ts":1684954854.7824316,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"79cdd708-4a5f-4695-8c92-589079cb7de0","kind":"ClusterPolicy","name":"restrict-nodeport"}
2023-05-24T19:00:54.784840036Z {"level":"info","ts":1684954854.7847354,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"7cd6db26-4942-4664-8c1f-1140b8b41ce2","kind":"ClusterPolicy","name":"restrict-sysctls"}
2023-05-24T19:00:54.791916987Z {"level":"info","ts":1684954854.7918308,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"231b2b37-e97e-47b7-864b-12eb6e27197c","kind":"ClusterPolicy","name":"disallow-latest-tag"}
2023-05-24T19:00:54.824848064Z {"level":"info","ts":1684954854.8247516,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"32e2eadf-5876-4027-9d92-99881bf1c7bc","kind":"ClusterPolicy","name":"disallow-privileged-containers"}
2023-05-24T19:00:55.050947453Z {"level":"info","ts":1684954855.0508196,"logger":"PolicyController","caller":"policy/policy_controller.go:171","msg":"policy created","uid":"506de046-8da4-48ae-8c98-77a0336153f5","kind":"ClusterPolicy","name":"require-non-root-groups"}

Slack discussion

No response

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.