ingress-nginx: Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: context deadline exceeded

Hi!

I’ve installed kubernetes cluster on three servers: l000d01kms001 (K8s master) and l000d01ksl001 & l000d01ksl001 - all basen Centos 7.5

Installed software:

docker-ce-cli-19.03.6-3.el7.x86_64 docker-ce-19.03.6-3.el7.x86_64 kubelet-1.17.3-0.x86_64 kubernetes-cni-0.7.5-0.x86_64 kubeadm-1.17.3-0.x86_64 kubectl-1.17.3-0.x86_64`

I’ve installed MetalLB (https://metallb.universe.tf) and then deployed Ingress-Nginx from URL https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/baremetal/deploy.yaml

When trying to deploy endpoint.yaml

apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: test-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules:

http: paths:

path: /testpath pathType: Prefix backend: serviceName: hellok8s-service servicePort: 8080

I’m getting error as below:

[root@l000d01kms001 KubeHelm]# kubectl apply -f endpoint.yaml --validate=false Error from server (InternalError): error when creating “endpoint.yaml”: Internal error occurred: failed calling webhook “validate.nginx.ingress.kubernetes.io”: Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: context deadline exceeded

`[root@l000d01kms001 KubeHelm]# kubectl logs pod/ingress-nginx-admission-create-44q78 -n ingress-nginx ^C [root@l000d01kms001 KubeHelm]# kubectl logs pod/ingress-nginx-admission-patch-gqwbr -n ingress-nginx {“level”:“info”,“msg”:“patching webhook configurations ‘ingress-nginx-admission’ mutating=false, validating=true, failurePolicy=Fail”,“source”:“k8s/k8s.go:38”,“time”:“2020-05-20T20:29:52Z”} {“level”:“info”,“msg”:“Patched hook(s)”,“source”:“k8s/k8s.go:91”,“time”:“2020-05-20T20:29:52Z”} [root@l000d01kms001 KubeHelm]# kubectl logs pod/ingress-nginx-controller-f8d756996-rhmjl -n ingress-nginx

NGINX Ingress controller Release: 0.32.0 Build: git-446845114 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.17.10

I0520 20:29:56.807345 6 flags.go:204] Watching for Ingress class: nginx W0520 20:29:56.807691 6 flags.go:249] SSL certificate chain completion is disabled (–enable-ssl-chain-completion=false) W0520 20:29:56.807752 6 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0520 20:29:56.807947 6 main.go:220] Creating API client for https://10.96.0.1:443 I0520 20:29:56.815151 6 main.go:264] Running in Kubernetes cluster version v1.17 (v1.17.3) - git (clean) commit 06ad960bfd03b39c8310aaf92d1e7c12ce618213 - platform linux/amd64 I0520 20:29:57.013592 6 main.go:105] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem I0520 20:29:57.018776 6 ssl.go:528] loading tls certificate from certificate path /usr/local/certificates/cert and key path /usr/local/certificates/key I0520 20:29:57.043694 6 nginx.go:263] Starting NGINX Ingress controller I0520 20:29:57.047238 6 event.go:278] Event(v1.ObjectReference{Kind:“ConfigMap”, Namespace:“ingress-nginx”, Name:“ingress-nginx-controller”, UID:“a204e9f1-6905-44c8-9fd0-e0aae4355e46”, APIVersion:“v1”, ResourceVersion:“2794479”, FieldPath:“”}): type: ‘Normal’ reason: ‘CREATE’ ConfigMap ingress-nginx/ingress-nginx-controller I0520 20:29:58.244113 6 nginx.go:307] Starting NGINX process I0520 20:29:58.244155 6 leaderelection.go:242] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx… I0520 20:29:58.244567 6 nginx.go:327] Starting validation webhook on :8443 with keys /usr/local/certificates/cert /usr/local/certificates/key I0520 20:29:58.244740 6 controller.go:139] Configuration changes detected, backend reload required. I0520 20:29:58.248112 6 leaderelection.go:252] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx I0520 20:29:58.248192 6 status.go:86] new leader elected: ingress-nginx-controller-f8d756996-rhmjl I0520 20:29:58.306048 6 controller.go:155] Backend successfully reloaded. I0520 20:29:58.306087 6 controller.go:164] Initial sync, sleeping for 1 second. [root@l000d01kms001 KubeHelm]#`

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 23 (7 by maintainers)

Most upvoted comments

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

+80

adamxy on Jun 23, 2020

I got the same error:

Error from server (InternalError): error when creating "ingressresource.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

after deleting the kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission now the ingress rule got created.

thirumurthis on Jul 4, 2020

@adamxy’s solution worked for me to, but why? What was wrong in my config? I’m also using MetalLB. Will this be a recurrent problem?

Regards,

marcboivin on Jul 10, 2020

Yeah, seems like an odd approach to just delete something.

We are using a private cluster on GKE. Not sure if related.

You can also disable the webhook: controller.admissionWebhooks.enabled.

I am still curious why this doesn’t work.

boxcee on Jul 16, 2020

@luppie when you deploy the ingress controller for the first time two Jobs are executed in order to generate the SSL certificate for the validation webhook. For this reason, it can take up to 90 seconds the start of the controller. After you start you can run the command

kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=90s

when that finish the ingress controller webhook is ready and you should not see that error

aledbf on May 20, 2020

@boxcee explanation here: kubernetes/kubernetes/issues/79739

msgongora on Aug 20, 2020

Hello there, Need help!!.

I have installed Nginx Ingress Controller (Not the NGINX Plus) from https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/ My Pods are running well Under NameSpace : nginx-ingress

Now when I am running the following yaml file getting error

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: template-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: learning.ingress
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx
                port:
                  number: 30816

Error from server (InternalError): error when creating “ingress_prashant.yaml”: Internal error occurred: failed calling webhook “validate.nginx.ingress.kubernetes.io”: Post “https://ingress-nginx-controller-admission.ingress.svc:443/networking/v1beta1/ingresses?timeout=10s”: service “ingress-nginx-controller-admission” not found

IF I remove the pathType: Prefix, then I am getting The Ingress “template-ingress” is invalid: spec.rules[0].http.paths[0].pathType: Required value: pathType must be specified

Can some one help me why webhook error comes, All youtube videos followed the https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/ and for them it worked well.

prashantkumarpathak on Aug 26, 2021

Run the ingress controller again worked for me.

Related issue : ( might help others ) I also had the issue of not able to access the localhost. So I reset Kubernetes from Docker Dashboard and then it worked.

hunzaboy on Oct 19, 2020

I’ve reinstalled Kubernetes with Calico network plugin and problem doesn’t occur.

luppie on May 22, 2020

@aledbf - this command returns `[root@l000d01kms001 KubeHelm]# kubectl wait --namespace ingress-nginx \

–for=condition=ready pod
–selector=app.kubernetes.io/component=controller
–timeout=90s pod/ingress-nginx-controller-f8d756996-rhmjl condition met [root@l000d01kms001 KubeHelm]#`

Error still occurs [root@l000d01kms001 KubeHelm]# kubectl apply -f endpoint.yaml --validate=false Error from server (InternalError): error when creating "endpoint.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: context deadline exceeded [root@l000d01kms001 KubeHelm]#

luppie on May 21, 2020