external-secrets: webhook CrashLoopBackOff

Helm chart v0.7.2, fresh deployment on GKE private cluster. No changes to helm chart.

NAME                                                READY   STATUS             RESTARTS        AGE
external-secrets-568c96645f-xxfvw                   1/1     Running            0               31m
external-secrets-cert-controller-67b8847f59-4mr6r   0/1     Running            0               31m
external-secrets-webhook-5874588dc-scl7p            0/1     CrashLoopBackOff   7 (3m57s ago)   31m

external-secrets-webhook-5874588dc-scl7p log

{"level":"info","ts":1675187014.5617464,"logger":"setup","msg":"validating certs"}
{"level":"error","ts":1675187014.5618227,"logger":"setup","msg":"invalid certs. retrying...","error":"stat /tmp/certs/tls.crt: no such file or directory","stacktrace":"github.com/external-secrets/external-secrets/cmd.waitForCerts\n\t/home/runner/work/external-secrets/external-secrets/cmd/webhook.go:188\ngithub.com/external-secrets/external-secrets/cmd.glob..func3\n\t/home/runner/work/external-secrets/external-secrets/cmd/webhook.go:82\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\ngithub.com/external-secrets/external-secrets/cmd.Execute\n\t/home/runner/work/external-secrets/external-secrets/cmd/root.go:225\nmain.main\n\t/home/runner/work/external-secrets/external-secrets/main.go:21\nruntime.main\n\t/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:250"}
{"level":"info","ts":1675187024.5630279,"logger":"setup","msg":"validating certs"}
{"level":"error","ts":1675187024.5631082,"logger":"setup","msg":"invalid certs. retrying...","error":"stat /tmp/certs/tls.crt: no such file or directory","stacktrace":"github.com/external-secrets/external-secrets/cmd.waitForCerts\n\t/home/runner/work/external-secrets/external-secrets/cmd/webhook.go:188\ngithub.com/external-secrets/external-secrets/cmd.glob..func3\n\t/home/runner/work/external-secrets/external-secrets/cmd/webhook.go:82\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\ngithub.com/external-secrets/external-secrets/cmd.Execute\n\t/home/runner/work/external-secrets/external-secrets/cmd/root.go:225\nmain.main\n\t/home/runner/work/external-secrets/external-secrets/main.go:21\nruntime.main\n\t/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:250"}
{"level":"info","ts":1675187034.563529,"logger":"setup","msg":"validating certs"}
{"level":"error","ts":1675187034.5636065,"logger":"setup","msg":"invalid certs. retrying...","error":"stat /tmp/certs/tls.crt: no such file or directory","stacktrace":"github.com/external-secrets/external-secrets/cmd.waitForCerts\n\t/home/runner/work/external-secrets/external-secrets/cmd/webhook.go:188\ngithub.com/external-secrets/external-secrets/cmd.glob..func3\n\t/home/runner/work/external-secrets/external-secrets/cmd/webhook.go:82\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\ngithub.com/external-secrets/external-secrets/cmd.Execute\n\t/home/runner/work/external-secrets/external-secrets/cmd/root.go:225\nmain.main\n\t/home/runner/work/external-secrets/external-secrets/main.go:21\nruntime.main\n\t/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:250"}
{"level":"error","ts":1675187044.5639164,"logger":"setup","msg":"unable to validate certificates","error":"context deadline exceeded","stacktrace":"github.com/external-secrets/external-secrets/cmd.glob..func3\n\t/home/runner/work/external-secrets/external-secrets/cmd/webhook.go:84\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\ngithub.com/external-secrets/external-secrets/cmd.Execute\n\t/home/runner/work/external-secrets/external-secrets/cmd/root.go:225\nmain.main\n\t/home/runner/work/external-secrets/external-secrets/main.go:21\nruntime.main\n\t/opt/hostedtoolcache/go/1.19.4/x64/src/runtime/proc.go:250"}

external-secrets-cert-controller-67b8847f59-4mr6r log

{"level":"error","ts":1675186740.7889977,"logger":"controllers.webhook-certs-updater","msg":"failed to inject conversion webhook","CustomResourceDefinition":"/secretstores.external-secrets.io","error":"secrets \"external-secrets-webhook\" not found","stacktrace":"github.com/external-secrets/external-secrets/pkg/controllers/crds.(*Reconciler).Reconcile\n\t/home/runner/work/external-secrets/external-secrets/pkg/controllers/crds/crds_controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":1675186740.789082,"msg":"Reconciler error","controller":"customresourcedefinition","controllerGroup":"apiextensions.k8s.io","controllerKind":"CustomResourceDefinition","CustomResourceDefinition":{"name":"secretstores.external-secrets.io"},"namespace":"","name":"secretstores.external-secrets.io","reconcileID":"b6d4c705-decd-4736-9468-b68bb66e2e67","error":"secrets \"external-secrets-webhook\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
E0131 17:39:00.793231       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"secretstore-validate.173f7445e2ef01b6", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ValidatingWebhookConfiguration", Namespace:"", Name:"secretstore-validate", UID:"331deed6-0182-4073-8165-2f0c23a3c1a2", APIVersion:"admissionregistration.k8s.io/v1", ResourceVersion:"165566522", FieldPath:""}, Reason:"UpdateFailed", Message:"secrets \"external-secrets-webhook\" not found", Source:v1.EventSource{Component:"validating-webhook-configuration", Host:""}, FirstTimestamp:time.Date(2023, time.January, 31, 17, 17, 9, 913993654, time.Local), LastTimestamp:time.Date(2023, time.January, 31, 17, 39, 0, 772704586, time.Local), Count:19, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'admission webhook "validation.gatekeeper.sh" denied the request: [manifest-must-have-approved-namespace] Manifest has a prohibited namespace [default]. Namespace other than default must be specified in manifest.' (will not retry!)
{"level":"error","ts":1675186740.7952096,"logger":"controllers.webhook-certs-updater","msg":"failed to inject conversion webhook","CustomResourceDefinition":"/externalsecrets.external-secrets.io","error":"secrets \"external-secrets-webhook\" not found","stacktrace":"github.com/external-secrets/external-secrets/pkg/controllers/crds.(*Reconciler).Reconcile\n\t/home/runner/work/external-secrets/external-secrets/pkg/controllers/crds/crds_controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":1675186740.7954235,"msg":"Reconciler error","controller":"customresourcedefinition","controllerGroup":"apiextensions.k8s.io","controllerKind":"CustomResourceDefinition","CustomResourceDefinition":{"name":"externalsecrets.external-secrets.io"},"namespace":"","name":"externalsecrets.external-secrets.io","reconcileID":"6577a4d2-1f52-43f9-b79a-6f5ee25f7958","error":"secrets \"external-secrets-webhook\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
E0131 17:39:00.809146       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"externalsecret-validate.173f7445e33983bb", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ValidatingWebhookConfiguration", Namespace:"", Name:"externalsecret-validate", UID:"922eea45-8dd1-45d9-a645-f14462ea507f", APIVersion:"admissionregistration.k8s.io/v1", ResourceVersion:"165566523", FieldPath:""}, Reason:"UpdateFailed", Message:"secrets \"external-secrets-webhook\" not found", Source:v1.EventSource{Component:"validating-webhook-configuration", Host:""}, FirstTimestamp:time.Date(2023, time.January, 31, 17, 17, 9, 918876603, time.Local), LastTimestamp:time.Date(2023, time.January, 31, 17, 39, 0, 784354218, time.Local), Count:19, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'admission webhook "validation.gatekeeper.sh" denied the request: [manifest-must-have-approved-namespace] Manifest has a prohibited namespace [default]. Namespace other than default must be specified in manifest.' (will not retry!)
{"level":"info","ts":1675186745.7385669,"logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{},{}]}
{"level":"info","ts":1675186750.3900604,"logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{},{}]}
{"level":"info","ts":1675186750.7385237,"logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{},{}]}
{"level":"info","ts":1675186755.7381608,"logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{},{}]}
{"level":"info","ts":1675186760.738423,"logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{},{}]}
{"level":"info","ts":1675186765.7382064,"logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{},{}]}
{"level":"info","ts":1675186770.738361,"logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{},{}]}
{"level":"info","ts":1675186775.7379782,"logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{},{}]}

It looks like cert-controller is waiting for "secrets \"external-secrets-webhook\" not found". Is it generated by webhook container? But webhook container log doesn’t tell much.