ingress-nginx: [v1] jettech/kube-webhook-certgen is not compatible with 1.22+

Tested helm chart 4.0.0-beta.1 with webhooks enabled

  admissionWebhooks:
    patch:
      image:
        image: rpkatz/kube-webhook-certgen
        tag: v1.5.2

and works perfectly, can create new ingress resources without problem.

Thanks.

/priority critical-urgent /assign Cc @tao12345666333 @strongjz before the v1 release this also should be fixed

I actually hit some further issues late yesterday with the admissionWebhooks.patch.enabled: false value applied. I have to sanitise this output slightly since it’s an internal deployment, the main difference being that it is an ingress backed by an actual service and using a rewrite rule. Hopefully I haven’t missed any other subtleties.

Applying a templatised YAML for one of my charts:

$ kubectl apply -f my-app.yaml
service/my-app created
deployment.apps/my-app created
pod/my-app-test-connection created
Error from server (InternalError): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.k8s.io/v1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{\"kubernetes.io/ingress.class\":\"nginx\",\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\",\"nginx.ingress.kubernetes.io/x-forwarded-prefix\":\"/my-app\"},\"labels\":{\"app.kubernetes.io/instance\":\"my-app\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"my-app\",\"app.kubernetes.io/version\":\"1.6.0\",\"helm.sh/chart\":\"my-app-1.2.0\"},\"name\":\"my-app\",\"namespace\":\"default\"},\"spec\":{\"rules\":[{\"host\":\"frontend.app.lan\",\"http\":{\"paths\":[{\"backend\":{\"service\":{\"name\":\"my-app\",\"port\":{\"number\":80}}},\"path\":\"/my-app(/|$)(.*)\",\"pathType\":\"Prefix\"}]}}]}}\n"}},"spec":{"rules":[{"host":"frontend.app.lan","http":{"paths":[{"backend":{"service":{"name":"my-app","port":{"number":80}}},"path":"/my-app(/|$)(.*)","pathType":"Prefix"}]}}]}}
to:
Resource: "networking.k8s.io/v1, Resource=ingresses", GroupVersionKind:
"networking.k8s.io/v1, Kind=Ingress"
Name: "my-app", Namespace: "default"
for: "my-app.yaml": Internal error occurred: failed calling webhook
"validate.nginx.ingress.kubernetes.io": Post
"https://ingress-nginx-controller-admission.kube-system.svc:443/networking/v1/ingresses?timeout=30s":
x509: certificate signed by unknown authority

That kind of makes sense I guess, since it’s likely that without the patching admission webhook, there might be some stubbed TLS certificate rather than the one which should have been minted. I may have to run through this stuff again to see if I’ve made some mistake.

@freshteapot

But the Changelog is still 3.34.

The chart’s changelog doesn’t appear to have been updated yet. Check the Chart.yaml instead. I use helm pull ingress-nginx/ingress-nginx --version <exact_version> and it pulls the one I asked for.

hello @maybe-sybr @kingdonb @fracarvic ,

We have released a new beta

% date ; helm search repo ingress --devel
Fri Aug 13 01:22:55 IST 2021
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                                                                                                                                
ingress-nginx/ingress-nginx     4.0.0-beta.2    1.0.0-beta.2    Ingress controller for Kubernetes using NGINX a...

This contains a new image for the certgen k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0@sha256:f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068

We are hoping you can test this beta.2 and provide feedback. Thanks, ; Long

Just tested helm chart 4.0.0-beta.2 with all of my customizations removed, with the included image and with re-enabled admission webhooks. It’s working for me. 👍

That helm install command appears to work fine for me, @rikatz. Output from the controller pod serving from a simple static landing page service for your reference:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"archive", BuildDate:"2021-03-30T00:00:00Z", GoVersion:"go1.16", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.0-beta.0", GitCommit:"a3f24e8459465495738af1b9cc6c3db80696e3c1", GitTreeState:"clean", BuildDate:"2021-06-22T21:00:26Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl logs -fn default           pod/ingress-nginx-controller-5977fdd7bd-hz6mn
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.0.0-beta.1
  Build:         da790570bd8d07d4980b175719f16c194301950d
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.20.1

-------------------------------------------------------------------------------

W0809 23:31:11.901950       2 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0809 23:31:11.904762       2 main.go:221] "Creating API client" host="https://10.0.0.1:443"
I0809 23:31:11.918946       2 main.go:265] "Running in Kubernetes cluster" major="1" minor="22+" git="v1.22.0-beta.0" state="clean" commit="a3f24e8459465495738af1b9cc6c3db80696e3c1" platform="linux/amd64"
I0809 23:31:12.163030       2 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0809 23:31:12.188577       2 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0809 23:31:12.209661       2 nginx.go:254] "Starting NGINX Ingress controller"
I0809 23:31:12.215795       2 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"default", Name:"ingress-nginx-controller", UID:"f10c0b6c-edd8-43ea-b421-29ccbb6fd27e", APIVersion:"v1", ResourceVersion:"90049", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap default/ingress-nginx-controller
I0809 23:31:13.312735       2 store.go:365] "Found valid IngressClass" ingress="myapp-dev/landing" ingressclass="nginx"
I0809 23:31:13.312842       2 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"myapp-dev", Name:"landing", UID:"99772a49-e575-4f7c-856a-aeb83b246533", APIVersion:"networking.k8s.io/v1", ResourceVersion:"89912", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0809 23:31:13.410102       2 leaderelection.go:243] attempting to acquire leader lease default/ingress-controller-leader...
I0809 23:31:13.410101       2 nginx.go:296] "Starting NGINX process"
I0809 23:31:13.411195       2 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0809 23:31:13.412183       2 controller.go:150] "Configuration changes detected, backend reload required"
I0809 23:31:13.414002       2 leaderelection.go:253] successfully acquired lease default/ingress-controller-leader
I0809 23:31:13.414033       2 status.go:84] "New leader elected" identity="ingress-nginx-controller-5977fdd7bd-hz6mn"
I0809 23:31:13.417211       2 status.go:284] "updating Ingress status" namespace="myapp-dev" ingress="landing" currentValue=[] newValue=[{IP:10.4.2.0 Hostname: Ports:[]}]
I0809 23:31:13.420441       2 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"myapp-dev", Name:"landing", UID:"99772a49-e575-4f7c-856a-aeb83b246533", APIVersion:"networking.k8s.io/v1", ResourceVersion:"90137", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0809 23:31:13.512673       2 controller.go:167] "Backend successfully reloaded"
I0809 23:31:13.512762       2 controller.go:178] "Initial sync, sleeping for 1 second"
I0809 23:31:13.512809       2 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"ingress-nginx-controller-5977fdd7bd-hz6mn", UID:"9f8ac8f1-3859-4360-8946-b984a804bfa5", APIVersion:"v1", ResourceVersion:"90086", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
10.88.0.1 - - [09/Aug/2021:23:34:26 +0000] "GET / HTTP/1.1" 200 509 "-" "curl/7.76.1" 89 0.001 [myapp-dev-landing-80] [] 10.88.1.88:80 509 0.000 200 7e352456ab148d039e774f4c059ec2dc

Thanks for chasing this up!

Folks, can you please test the webhook with image:

rpkatz/kube-webhook-certgen:v1.5.2

And check if the problem persists? If this is solved, I’m going to submit a PR to the original project

In my case ValidatingWebhookConfiguration stayed installed from previous helm installation. I deleted mine with

 kubectl delete ValidatingWebhookConfiguration ingress-nginx-admission

and install 4.0.0 helm chart with controller.admissionWebhooks.enabled: false and can create new ingress resources without errors.