keda: Setting failurePolicy to Fail in the admission webhook does not work

Report

We’ve installed keda via the downloaded release yaml.

We want to use the admission webhook with “failurePolicy: Fail”. As soon as we change that we see the following issue, when fluxcd tries to apply a namesapce using a scaledobject.

{"level":"error","ts":"2023-11-15T13:11:16.956Z","msg":"Reconciliation failed after 2.426684853s, next try in 5m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"gatekeeper","namespace":"flux-system"},"namespace":"flux-system","name":"gatekeeper","reconcileID":"9d4d6fe1-da59-40e9-af8c-798651024822","revision":"dev@sha1:b7ced8d1a6d67107cedfb8bdd665d81de2877ba0","error":"ScaledObject/gatekeeper/httpcache dry-run failed, reason: InternalError: Internal error occurred: failed calling webhook \"vscaledobject.kb.io\": failed to call webhook: Post \"[https://keda-admission-webhooks.keda.svc:443/validate-keda-sh-v1alpha1-scaledobject?timeout=10s](https://keda-admission-webhooks.keda.svc/validate-keda-sh-v1alpha1-scaledobject?timeout=10s)\": EOF\n"}

The admisson webhook deployment is running and the svc is reachable via port-forward. The used certificates use the right names.

How can we further debug this?

I’m not sure if “internal error” is an error of the admission controller or if the admission controller can’t be reached. The admission controller itself does not log any error.

For us it looks like the admission webhook does not work at all but the error is ignored with the default config?

Expected Behavior

The admission webhook works with: failurePolicy: Fail

Actual Behavior

The admission webhook can’t be used becauser of an “internal error”.

Steps to Reproduce the Problem

1. edit the admisison controller and set failurePolicy: Fail

Logs from KEDA operator

The only error i found in the operator logs was:

2023-11-15T13:42:32Z	ERROR	cert-rotation	Webhook not found. Unable to update certificate.	{"name": "keda-admission-webhooks", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "error": "ValidatingWebhookConfiguration.admissionregistration.k8s.io \"keda-admission-webhooks\" not found"}

So the name of the webhook seemed not to match.

Changing the name of the validating webhook from keda-admission to keda-admission-webhooks did not help.

Afterwards we saw errors like:

2023-11-15T14:07:30Z	ERROR	cert-rotation	Error updating webhook with certificate	{"name": "keda-admission-webhooks", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "error": "Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io \"keda-admission-webhooks\": the object has been modified; please apply your changes to the latest version and try again"}

KEDA Version

2.12.0

Kubernetes Version

1.26

Platform

Microsoft Azure

Scaler Details

No response

Anything else?

No response