ingress-nginx: apply ingress rule error after install ingress-nginx: x509 certificate is not valid ingress-nginx-controller-admission.ingress-nginx.svc

My fix for this issue without deleting the validatingwebhookconfigurations. Inspired by https://github.com/kubernetes/ingress-nginx/issues/5968#issuecomment-700287814

CA=$(kubectl -n ingress-nginx get secret ingress-nginx-admission -ojsonpath='{.data.ca}')
kubectl patch validatingwebhookconfigurations ingress-nginx-admission --type='json' -p='[{"op": "add", "path": "/webhooks/0/clientConfig/caBundle", "value":"'$CA'"}]'

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

My fix for this issue without deleting the validatingwebhookconfigurations. Inspired by #5968 (comment)

CA=$(kubectl -n ingress-nginx get secret ingress-nginx-admission -ojsonpath='{.data.ca}')
kubectl patch validatingwebhookconfigurations ingress-nginx-admission --type='json' -p='[{"op": "add", "path": "/webhooks/0/clientConfig/caBundle", "value":"'$CA'"}]'

I am seeing this problem also:

Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.kube-system.svc:443/extensions/v1beta1/ingresses?timeout=30s": x509: certificate signed by unknown authority

This is in minikube with Ingress-nginx installed using minikube addons enable ingress

I can ‘resolve’ it using kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission but this does not seem like a solution, its just a hack to get it working.

Borrowing from answers above, here’s what resolved this for me. Issue started when I deleted an entire namespace that had contained nginx-ingress, and then tried to reinstall everything via helm. Kept getting upon helm upgrade --install:

Error: UPGRADE FAILED: failed to create resource: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://foobar-ingress-nginx-controller-admission.foobar.svc:443/networking/v1beta1/ingresses?timeout=10s": x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "nil1")

Solution:

# Find name of the ingress-nginx-admission resource
kubectl get -A ValidatingWebhookConfiguration
# Delete it
kubectl delete -A ValidatingWebhookConfiguration <name>
# Example:
kubectl delete -A ValidatingWebhookConfiguration foobar-ingress-nginx-admission

@liminghua999 can you please check if the output of these two commands match?

$ kubectl -n ingress-nginx get validatingwebhookconfigurations ingress-nginx-admission -ojsonpath='{.webhooks[0].clientConfig.caBundle}'

$ kubectl -n ingress-nginx get secret ingress-nginx-admission -ojsonpath='{.data.ca}'

Should we reach a conclusion about the root cause? and how can this be avoided in the future?

I’m encountering this issue, and I don’t have a valid workaround, because my ingress-nginx is deployed with fluxcd, so every time I delete the ValidatingWebhookConfiguration, it is recreated.

I am not sure if it’s related, but I followed @costela’s instructions, and this command has no output:

kubectl -n ingress-nginx get validatingwebhookconfigurations ingress-nginx-admission -ojsonpath='{.webhooks[0].clientConfig.caBundle}'

This command

kubectl -n ingress-nginx get validatingwebhookconfigurations ingress-nginx-admission -ojsonpath='{.webhooks[0].clientConfig}'

shows

{"service":{"name":"ingress-nginx-controller-admission","namespace":"ingress-nginx","path":"/networking/v1beta1/ingresses","port":443}}

Is this the problem, no caBundle? Can someone help solve this?

Or is there a workaround to delete the ValidatingWebhookConfiguration from the source yaml, which I took from here? Can someone explain how exactly to do that?

My fix for this issue without deleting the validatingwebhookconfigurations. Inspired by #5968 (comment)

CA=$(kubectl -n ingress-nginx get secret ingress-nginx-admission -ojsonpath='{.data.ca}')
kubectl patch validatingwebhookconfigurations ingress-nginx-admission --type='json' -p='[{"op": "add", "path": "/webhooks/0/clientConfig/caBundle", "value":"'$CA'"}]'

I faced the same issue after having kille the Ingress controller pod. When it got recreated a deployed Jaeger operator complaind not being able to call the ingress-nginx-controller-admission webhook on port 443. After I applied the patch, all was fine again.

Hi

I’ve having the same problem. And based in the comment

My fix for this issue without deleting the validatingwebhookconfigurations. Inspired by #5968 (comment)

CA=$(kubectl -n ingress-nginx get secret ingress-nginx-admission -ojsonpath='{.data.ca}')
kubectl patch validatingwebhookconfigurations ingress-nginx-admission --type='json' -p='[{"op": "add", "path": "/webhooks/0/clientConfig/caBundle", "value":"'$CA'"}]'

I’ve been investigating. I don’t use Helm, and I’ve had to configure each component individually.

There’re two Jobs for nginx-admission-webhooks, create and patch. And Based in the Image’s documentation we have a command line with their especifications:

https://github.com/jet/kube-webhook-certgen#patch

  --patch-validating              If true, patch validatingwebhookconfiguration (default true)

Doing that I’ve got a correct validatingwebhookconfiguration config.

I hope that help you.

For anyone banging their heads against this:

TLDR: Metallb/Loadbalancer was not running/working correctly causing the service not to be available. TLS/x509errors threw me on a wild goose chase.

My only indication was that the service created by nginx-ingress was not put in a ready state ( nothing apparent except ArgoCD saying it wasnt ). When investigating i saw that i had a config error in my metallb chart values. After fixing that the service finally became available, all errors disappeared and i could create ingresses again. (be sure to terminate sync-action in flux/argo if applicable and retry sync)

Inspired by @marv254 's comment.

I had a similar issue with missing caBundle in the validatingwebhookconfigurations/ingress-nginx-admission, and it was installed via ArgoCD and a customized ingress-nginx helm chart.

Since ingress-nginx-admission-patch has an annotation "helm.sh/hook": post-install, caBundle will only appear in the validatingwebhookconfigurations after the Chart is fully installed.

In my case, my custom chart contains some resources that required the admission webhook with the caBundle, and that caused Helm chart to never reaches the ‘post-install’ stage.

My solution is to add the annotation "helm.sh/hook": post-install to my custom resource as well.

I ran into the the same problem. I fixed it by deleting and reapplying the ingress-nginx-admission-patch job.