kyverno: kyverno > v1.5.1 on gke causes a "Timeout registering admission control webhook "

we are trying to get kyverno running on gke.

we are currently installing kyverno from https://raw.githubusercontent.com/kyverno/kyverno/release-1.5/definitions/release/install.yaml.

the current version fails to startup with a timeout registering the admission control webhook. this does not happen when manually editing the install.yaml and changing the image to a version < v.1.5.2.

for example image: ghcr.io/kyverno/kyverno:v1.5.1 works without any issues.

this is the startup log of the failing version:

I0110 08:41:43.961703       1 version.go:17]  "msg"="Kyverno"  "Version"="v1.5.3-3-g30eec81b"
I0110 08:41:43.961738       1 version.go:18]  "msg"="Kyverno"  "BuildHash"="(HEAD/30eec81b639fa24ed39c1c9b59bcb02db2a2c748"
I0110 08:41:43.961757       1 version.go:19]  "msg"="Kyverno"  "BuildTime"="2022-01-07_10:16:33PM"
I0110 08:41:43.962397       1 config.go:104] CreateClientConfig "msg"="Using in-cluster configuration"
I0110 08:41:45.165668       1 request.go:665] Waited for 1.174493721s due to client-side throttling, not priority and fairness, request: GET:https://10.13.8.1:443/apis/storage.cnrm.cloud.google.com/v1beta1?timeout=32s
I0110 08:41:45.567650       1 util.go:97]  "msg"="CRD found"  "gvr"="kyverno.io/v1, Resource=clusterpolicies"
I0110 08:41:45.568701       1 util.go:97]  "msg"="CRD found"  "gvr"="wgpolicyk8s.io/v1alpha2, Resource=clusterpolicyreports"
I0110 08:41:45.569741       1 util.go:97]  "msg"="CRD found"  "gvr"="wgpolicyk8s.io/v1alpha2, Resource=policyreports"
I0110 08:41:45.570784       1 util.go:97]  "msg"="CRD found"  "gvr"="kyverno.io/v1alpha2, Resource=clusterreportchangerequests"
I0110 08:41:45.571685       1 util.go:97]  "msg"="CRD found"  "gvr"="kyverno.io/v1alpha2, Resource=reportchangerequests"
I0110 08:41:45.987990       1 dynamicconfig.go:148] ConfigData "msg"="init configuration from commandline arguments for filterK8sResources"
I0110 08:41:45.988368       1 dynamicconfig.go:328] ConfigData "msg"="Init resource filters"  "filters"=[{"Kind":"Event","Namespace":"*","Name":"*"},{"Kind":"*","Namespace":"kube-system","Name":"*"},{"Kind":"*","Namespace":"kube-public","Name":"*"},{"Kind":"*","Namespace":"kube-node-lease","Name":"*"},{"Kind":"Node","Namespace":"*","Name":"*"},{"Kind":"APIService","Namespace":"*","Name":"*"},{"Kind":"TokenReview","Namespace":"*","Name":"*"},{"Kind":"SubjectAccessReview","Namespace":"*","Name":"*"},{"Kind":"*","Namespace":"kyverno","Name":"*"},{"Kind":"Binding","Namespace":"*","Name":"*"},{"Kind":"ReplicaSet","Namespace":"*","Name":"*"},{"Kind":"ReportChangeRequest","Namespace":"*","Name":"*"},{"Kind":"ClusterReportChangeRequest","Namespace":"*","Name":"*"},{"Kind":"PolicyReport","Namespace":"*","Name":"*"},{"Kind":"ClusterPolicyReport","Namespace":"*","Name":"*"}]
I0110 08:41:45.988402       1 dynamicconfig.go:339] ConfigData "msg"="Init resource "  "excludeRoles"=""
I0110 08:41:45.995249       1 metrics.go:139] MetricsConfig "msg"="Configuring metrics refresh at a periodic rate of 24h0m0s"
I0110 08:41:45.995469       1 deleg.go:130] setup "msg"="enabling metrics service"  "address"=":8000"
I0110 08:41:45.997555       1 leaderelection.go:248] attempting to acquire leader lease kyverno/webhook-register...
I0110 08:41:46.032940       1 leaderelection.go:258] successfully acquired lease kyverno/webhook-register
I0110 08:41:46.033316       1 leaderelection.go:94] webhookRegister/LeaderElection "msg"="started leading" "id"="kyverno-84dcc9b4d4-sb96v_b650112d-ad0c-4e06-aedd-3b2ca169b5fc"
I0110 08:41:46.103229       1 certRenewer.go:80] CertRenewer/InitTLSPemPair "msg"="using existing TLS key/certificate pair"
E0110 08:41:46.187351       1 registration.go:227] Register/ValidateWebhookConfigurations "msg"="unable to fetch ConfigMap" "error"="configmaps \"init-config\" not found"  "name"="init-config" "namespace"="kyverno"
I0110 08:41:48.364850       1 request.go:665] Waited for 1.171198821s due to client-side throttling, not priority and fairness, request: GET:https://10.13.8.1:443/apis/autoscaling/v1?timeout=32s
I0110 08:41:49.365107       1 request.go:665] Waited for 2.171413958s due to client-side throttling, not priority and fairness, request: GET:https://10.13.8.1:443/apis/certificates.k8s.io/v1?timeout=32s
I0110 08:41:50.564999       1 request.go:665] Waited for 3.371247744s due to client-side throttling, not priority and fairness, request: GET:https://10.13.8.1:443/apis/policy/v1beta1?timeout=32s
I0110 08:41:51.764891       1 request.go:665] Waited for 4.571072314s due to client-side throttling, not priority and fairness, request: GET:https://10.13.8.1:443/apis/admissionregistration.k8s.io/v1beta1?timeout=32s
[otta@kubectl ~]$ kubectl logs -f kyverno-84dcc9b4d4-sb96v -n kyverno  | grep -v "due to client-side throttling"
I0110 08:41:43.961703       1 version.go:17]  "msg"="Kyverno"  "Version"="v1.5.3-3-g30eec81b"
I0110 08:41:43.961738       1 version.go:18]  "msg"="Kyverno"  "BuildHash"="(HEAD/30eec81b639fa24ed39c1c9b59bcb02db2a2c748"
I0110 08:41:43.961757       1 version.go:19]  "msg"="Kyverno"  "BuildTime"="2022-01-07_10:16:33PM"
I0110 08:41:43.962397       1 config.go:104] CreateClientConfig "msg"="Using in-cluster configuration"
I0110 08:41:45.567650       1 util.go:97]  "msg"="CRD found"  "gvr"="kyverno.io/v1, Resource=clusterpolicies"
I0110 08:41:45.568701       1 util.go:97]  "msg"="CRD found"  "gvr"="wgpolicyk8s.io/v1alpha2, Resource=clusterpolicyreports"
I0110 08:41:45.569741       1 util.go:97]  "msg"="CRD found"  "gvr"="wgpolicyk8s.io/v1alpha2, Resource=policyreports"
I0110 08:41:45.570784       1 util.go:97]  "msg"="CRD found"  "gvr"="kyverno.io/v1alpha2, Resource=clusterreportchangerequests"
I0110 08:41:45.571685       1 util.go:97]  "msg"="CRD found"  "gvr"="kyverno.io/v1alpha2, Resource=reportchangerequests"
I0110 08:41:45.987990       1 dynamicconfig.go:148] ConfigData "msg"="init configuration from commandline arguments for filterK8sResources"
I0110 08:41:45.988368       1 dynamicconfig.go:328] ConfigData "msg"="Init resource filters"  "filters"=[{"Kind":"Event","Namespace":"*","Name":"*"},{"Kind":"*","Namespace":"kube-system","Name":"*"},{"Kind":"*","Namespace":"kube-public","Name":"*"},{"Kind":"*","Namespace":"kube-node-lease","Name":"*"},{"Kind":"Node","Namespace":"*","Name":"*"},{"Kind":"APIService","Namespace":"*","Name":"*"},{"Kind":"TokenReview","Namespace":"*","Name":"*"},{"Kind":"SubjectAccessReview","Namespace":"*","Name":"*"},{"Kind":"*","Namespace":"kyverno","Name":"*"},{"Kind":"Binding","Namespace":"*","Name":"*"},{"Kind":"ReplicaSet","Namespace":"*","Name":"*"},{"Kind":"ReportChangeRequest","Namespace":"*","Name":"*"},{"Kind":"ClusterReportChangeRequest","Namespace":"*","Name":"*"},{"Kind":"PolicyReport","Namespace":"*","Name":"*"},{"Kind":"ClusterPolicyReport","Namespace":"*","Name":"*"}]
I0110 08:41:45.988402       1 dynamicconfig.go:339] ConfigData "msg"="Init resource "  "excludeRoles"=""
I0110 08:41:45.995249       1 metrics.go:139] MetricsConfig "msg"="Configuring metrics refresh at a periodic rate of 24h0m0s"
I0110 08:41:45.995469       1 deleg.go:130] setup "msg"="enabling metrics service"  "address"=":8000"
I0110 08:41:45.997555       1 leaderelection.go:248] attempting to acquire leader lease kyverno/webhook-register...
I0110 08:41:46.032940       1 leaderelection.go:258] successfully acquired lease kyverno/webhook-register
I0110 08:41:46.033316       1 leaderelection.go:94] webhookRegister/LeaderElection "msg"="started leading" "id"="kyverno-84dcc9b4d4-sb96v_b650112d-ad0c-4e06-aedd-3b2ca169b5fc"
I0110 08:41:46.103229       1 certRenewer.go:80] CertRenewer/InitTLSPemPair "msg"="using existing TLS key/certificate pair"
E0110 08:41:46.187351       1 registration.go:227] Register/ValidateWebhookConfigurations "msg"="unable to fetch ConfigMap" "error"="configmaps \"init-config\" not found"  "name"="init-config" "namespace"="kyverno"
I0110 08:42:29.178044       1 certmanager.go:107] CertManager "msg"="read TLS pem pair from the secret"
I0110 08:42:29.180435       1 leaderelection.go:248] attempting to acquire leader lease kyverno/kyverno...
I0110 08:42:29.181665       1 reportrequest.go:178] ReportChangeRequestGenerator "msg"="start"
I0110 08:42:29.181731       1 controller.go:118] EventGenerator "msg"="start"
I0110 08:42:29.181800       1 informer.go:109] PolicyCacheController "msg"="starting"
I0110 08:42:29.245868       1 leaderelection.go:113] kyverno/LeaderElection "msg"="another instance has been elected as leader" "current id"="kyverno-84dcc9b4d4-sb96v_bf7513cc-e9f9-4260-a5a3-1f0db1b524ac" "leader"="kyverno-84dcc9b4d4-sb96v_4ce65b47-e425-425c-b885-1043dc5af8a0"
E0110 08:42:29.361632       1 deleg.go:144] setup "msg"="Timeout registering admission control webhooks" "error"=null

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 21 (7 by maintainers)

Most upvoted comments

–webhookRegistrationTimeout=120

Is what works for me.

On Fri, Jan 14, 2022, 6:48 AM Marc @.***> wrote:

Could you clarify, is it --webhookTimeout or --webhookRegistrationTimeout (it looks like the latter in the PR?)

— Reply to this email directly, view it on GitHub https://github.com/kyverno/kyverno/issues/2951#issuecomment-1013132643, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKXLCM2ZEDRKM32ZP37Z4TUWASTVANCNFSM5LTEZ6TA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

dddomin3 on Jan 14, 2022

It fixed my issue, not sure about the original poster though. I think they’re on a different version, but it should give them hope.

On Fri, Jan 14, 2022, 12:12 AM shuting @.***> wrote:

should it be changing the time out at the end of these requests? Because it isn’t for me @realshuting https://github.com/realshuting

No, it’s the timeout for webhook registration

I0113 22:17:58.943575 1 registration.go:640] Register “msg”=“Endpoint ready” “name”=“kyverno-svc” “ns”=“kyverno-system”

I can see the Endpoint was ready, did Kyverno back to running once your fixed this flag - “–webhookTimeout=120” ?

— Reply to this email directly, view it on GitHub https://github.com/kyverno/kyverno/issues/2951#issuecomment-1012835305, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKXLCOE6FWJFUZHHSCYUR3UV7EEXANCNFSM5LTEZ6TA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

dddomin3 on Jan 14, 2022