ingress-nginx: k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress

Hi @sysnet4admin

So for ingress v1, some things have changed with Ingress Class:

• If you have only one ingress controller running in your environment and you still want to use ingress class, you should add the annotation “ingressclass.kubernetes.io/is-default-class” in your ingress class, so new Ingress objects will have this one as defaulted
• If you don’t care about ingress class, or you have a lot of Ingress objects without ingress class configuration, you can run the ingress-controller with the option --watch-ingress-without-class=true Can you please make a new attempt and let us know?

Thanks!

I am on Minikube version 1.23 and am seeing the same problem. Should I still need to use the beta version?

1st options is worked on properly too. but it should(?) remove ValidatingWebhookConfiguration ingress-nginx-admission to apply ingress backend configuration.

Here is the result of 1st option.

Check before run

[root@m-k8s ~]# cat  deploy.yaml | grep ingressclass.kubernetes.io/is-default-class
    ingressclass.kubernetes.io/is-default-class: "true"

Actual run

[root@m-k8s ~]# k get po -n ingress-nginx
NAME                                        READY   STATUS             RESTARTS      AGE
ingress-nginx-admission-create--1-m4f8v     0/1     Completed          0             2m
ingress-nginx-admission-patch--1-5tkp2      0/1     CrashLoopBackOff   4 (15s ago)   2m
ingress-nginx-controller-7f8cb86456-dz6p5   1/1     Running            0             2m
[root@m-k8s ~]# k logs -n ingress-nginx ingress-nginx-controller-7f8cb86456-dz6p5
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.0.0-beta.1
  Build:         da790570bd8d07d4980b175719f16c194301950d
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.20.1

-------------------------------------------------------------------------------

W0806 13:44:21.151566       6 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0806 13:44:21.156023       6 main.go:221] "Creating API client" host="https://10.96.0.1:443"
I0806 13:44:21.227695       6 main.go:265] "Running in Kubernetes cluster" major="1" minor="22" git="v1.22.0" state="clean" commit="c2b5237ccd9c0f1d600d3072634ca66cefdf272f" platform="linux/amd64"
I0806 13:44:21.795035       6 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0806 13:44:21.880159       6 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0806 13:44:21.924233       6 nginx.go:254] "Starting NGINX Ingress controller"
I0806 13:44:21.971117       6 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"79f6f68c-5d6a-439b-be4f-a2e3e16fbc49", APIVersion:"v1", ResourceVersion:"104171", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0806 13:44:23.053408       6 store.go:361] "Ignoring ingress because of error while validating ingress class" ingress="default/nginx-ingress" error="ingress does not contain a valid IngressClass"
I0806 13:44:23.126723       6 nginx.go:296] "Starting NGINX process"
I0806 13:44:23.127932       6 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0806 13:44:23.128563       6 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I0806 13:44:23.130649       6 controller.go:150] "Configuration changes detected, backend reload required"
I0806 13:44:23.174171       6 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader
I0806 13:44:23.175632       6 status.go:84] "New leader elected" identity="ingress-nginx-controller-7f8cb86456-dz6p5"
I0806 13:44:23.255121       6 status.go:204] "POD is not ready" pod="ingress-nginx/ingress-nginx-controller-7f8cb86456-dz6p5" node="w2-k8s"
I0806 13:44:23.404692       6 controller.go:167] "Backend successfully reloaded"
I0806 13:44:23.404790       6 controller.go:178] "Initial sync, sleeping for 1 second"
I0806 13:44:23.407392       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7f8cb86456-dz6p5", UID:"9d6059ef-7a9a-4c5c-a897-bbc49b7d3e38", APIVersion:"v1", ResourceVersion:"104261", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
[root@m-k8s ~]# ka  ingress.yaml
Error from server (InternalError): error when creating "_Lecture_k8s_learning.kit/ch4/4.8/ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": x509: certificate signed by unknown authority
[root@m-k8s ~]# kubectl delete ValidatingWebhookConfiguration ingress-nginx-admission
validatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-admission" deleted
[root@m-k8s ~]# ka  ingress.yaml
ingress.networking.k8s.io/nginx-ingress created
[root@m-k8s ~]# k logs -n ingress-nginx ingress-nginx-controller-7f8cb86456-dz6p5
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.0.0-beta.1
  Build:         da790570bd8d07d4980b175719f16c194301950d
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.20.1

-------------------------------------------------------------------------------

W0806 13:44:21.151566       6 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0806 13:44:21.156023       6 main.go:221] "Creating API client" host="https://10.96.0.1:443"
I0806 13:44:21.227695       6 main.go:265] "Running in Kubernetes cluster" major="1" minor="22" git="v1.22.0" state="clean" commit="c2b5237ccd9c0f1d600d3072634ca66cefdf272f" platform="linux/amd64"
I0806 13:44:21.795035       6 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0806 13:44:21.880159       6 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0806 13:44:21.924233       6 nginx.go:254] "Starting NGINX Ingress controller"
I0806 13:44:21.971117       6 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"79f6f68c-5d6a-439b-be4f-a2e3e16fbc49", APIVersion:"v1", ResourceVersion:"104171", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0806 13:44:23.053408       6 store.go:361] "Ignoring ingress because of error while validating ingress class" ingress="default/nginx-ingress" error="ingress does not contain a valid IngressClass"
I0806 13:44:23.126723       6 nginx.go:296] "Starting NGINX process"
I0806 13:44:23.127932       6 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0806 13:44:23.128563       6 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I0806 13:44:23.130649       6 controller.go:150] "Configuration changes detected, backend reload required"
I0806 13:44:23.174171       6 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader
I0806 13:44:23.175632       6 status.go:84] "New leader elected" identity="ingress-nginx-controller-7f8cb86456-dz6p5"
I0806 13:44:23.255121       6 status.go:204] "POD is not ready" pod="ingress-nginx/ingress-nginx-controller-7f8cb86456-dz6p5" node="w2-k8s"
I0806 13:44:23.404692       6 controller.go:167] "Backend successfully reloaded"
I0806 13:44:23.404790       6 controller.go:178] "Initial sync, sleeping for 1 second"
I0806 13:44:23.407392       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7f8cb86456-dz6p5", UID:"9d6059ef-7a9a-4c5c-a897-bbc49b7d3e38", APIVersion:"v1", ResourceVersion:"104261", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0806 13:48:17.343387       6 store.go:336] "Ignoring ingress because of error while validating ingress class" ingress="default/nginx-ingress" error="ingress does not contain a valid IngressClass"
2021/08/06 13:48:26 http: TLS handshake error from 172.16.171.64:32934: remote error: tls: bad certificate
I0806 13:49:11.736044       6 store.go:365] "Found valid IngressClass" ingress="default/nginx-ingress" ingressclass="nginx"
W0806 13:49:11.737919       6 controller.go:912] Error obtaining Endpoints for Service "default/ing-hn": no object matching key "default/ing-hn" in local store
I0806 13:49:11.738426       6 controller.go:150] "Configuration changes detected, backend reload required"
I0806 13:49:11.744970       6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx-ingress", UID:"4a18b17d-10b9-4704-89be-2259805c26ef", APIVersion:"networking.k8s.io/v1", ResourceVersion:"104781", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0806 13:49:11.847534       6 controller.go:167] "Backend successfully reloaded"
I0806 13:49:11.849659       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7f8cb86456-dz6p5", UID:"9d6059ef-7a9a-4c5c-a897-bbc49b7d3e38", APIVersion:"v1", ResourceVersion:"104261", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0806 13:49:23.210850       6 status.go:284] "updating Ingress status" namespace="default" ingress="nginx-ingress" currentValue=[] newValue=[{IP:192.168.1.102 Hostname: Ports:[]}]
W0806 13:49:23.223796       6 controller.go:912] Error obtaining Endpoints for Service "default/ing-hn": no object matching key "default/ing-hn" in local store
I0806 13:49:23.225543       6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx-ingress", UID:"4a18b17d-10b9-4704-89be-2259805c26ef", APIVersion:"networking.k8s.io/v1", ResourceVersion:"104804", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
[root@m-k8s ~]# k get ingress
NAME            CLASS   HOSTS   ADDRESS         PORTS   AGE
nginx-ingress   nginx   *       192.168.1.102   80      39s
[root@m-k8s ~]# curl 192.168.1.101:31603
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

@rikatz I appropriate your help. It solve by 2nd option and I aim to try 1st option due to my curiosity 😃 Here is the result of 2nd option.

[root@m-k8s ~]# k get po -n ingress-nginx
NAME                                        READY   STATUS      RESTARTS      AGE
ingress-nginx-admission-create--1-6hlqd     0/1     Completed   0             63s
ingress-nginx-admission-patch--1-qjhld      0/1     Error       3 (40s ago)   63s
ingress-nginx-controller-64dd9bfd84-df622   1/1     Running     0             64s
[root@m-k8s ~]# k logs -n ingress-nginx ingress-nginx-controller-64dd9bfd84-df622
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.0.0-beta.1
  Build:         da790570bd8d07d4980b175719f16c194301950d
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.20.1

-------------------------------------------------------------------------------

W0806 13:30:24.461379       8 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0806 13:30:24.462161       8 main.go:221] "Creating API client" host="https://10.96.0.1:443"
I0806 13:30:24.535573       8 main.go:265] "Running in Kubernetes cluster" major="1" minor="22" git="v1.22.0" state="clean" commit="c2b5237ccd9c0f1d600d3072634ca66cefdf272f" platform="linux/amd64"
I0806 13:30:25.486555       8 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0806 13:30:25.557902       8 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0806 13:30:25.614048       8 nginx.go:254] "Starting NGINX Ingress controller"
I0806 13:30:25.655365       8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"b2807e0e-6e43-4eaf-80f2-268629cee9f3", APIVersion:"v1", ResourceVersion:"102630", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0806 13:30:26.730882       8 store.go:365] "Found valid IngressClass" ingress="default/nginx-ingress" ingressclass="_"
I0806 13:30:26.735768       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx-ingress", UID:"8aca632d-d1ce-43d3-a553-972d3011cb78", APIVersion:"networking.k8s.io/v1", ResourceVersion:"94709", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0806 13:30:26.815910       8 nginx.go:296] "Starting NGINX process"
I0806 13:30:26.816824       8 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I0806 13:30:26.817134       8 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
W0806 13:30:26.819656       8 controller.go:912] Error obtaining Endpoints for Service "default/ing-hn": no object matching key "default/ing-hn" in local store
I0806 13:30:26.820225       8 controller.go:150] "Configuration changes detected, backend reload required"
I0806 13:30:26.861440       8 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader
I0806 13:30:26.884775       8 status.go:84] "New leader elected" identity="ingress-nginx-controller-64dd9bfd84-df622"
I0806 13:30:26.971081       8 status.go:204] "POD is not ready" pod="ingress-nginx/ingress-nginx-controller-64dd9bfd84-df622" node="w2-k8s"
I0806 13:30:27.099884       8 controller.go:167] "Backend successfully reloaded"
I0806 13:30:27.099967       8 controller.go:178] "Initial sync, sleeping for 1 second"
I0806 13:30:27.101717       8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-64dd9bfd84-df622", UID:"e5a16488-9fb4-406f-97b7-79b52620c506", APIVersion:"v1", ResourceVersion:"102731", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W0806 13:30:30.639407       8 controller.go:912] Error obtaining Endpoints for Service "default/ing-hn": no object matching key "default/ing-hn" in local store
W0806 13:30:39.472080       8 controller.go:912] Error obtaining Endpoints for Service "default/ing-hn": no object matching key "default/ing-hn" in local store
W0806 13:30:42.805819       8 controller.go:912] Error obtaining Endpoints for Service "default/ing-hn": no object matching key "default/ing-hn" in local store
I0806 13:31:27.031656       8 status.go:284] "updating Ingress status" namespace="default" ingress="nginx-ingress" currentValue=[] newValue=[{IP:192.168.1.102 Hostname: Ports:[]}]
W0806 13:31:27.047295       8 controller.go:912] Error obtaining Endpoints for Service "default/ing-hn": no object matching key "default/ing-hn" in local store
I0806 13:31:27.048271       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx-ingress", UID:"8aca632d-d1ce-43d3-a553-972d3011cb78", APIVersion:"networking.k8s.io/v1", ResourceVersion:"102865", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
[root@m-k8s ~]# k get ingress
NAME            CLASS    HOSTS   ADDRESS         PORTS   AGE
nginx-ingress   <none>   *       192.168.1.102   80      83m
[root@m-k8s ~]# k get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.107.114.168   <none>        80:32242/TCP,443:32725/TCP   2m18s
ingress-nginx-controller-admission   ClusterIP   10.107.66.204    <none>        443/TCP                      2m18s
[root@m-k8s ~]# curl 192.168.1.102:32242
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@m-k8s ~]# curl 192.168.1.101:32242
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@m-k8s ~]# curl 192.168.1.101:32242/ip
request_method : GET | ip_dest: 172.16.103.147

Ok. Can you try this one and give feedback. https://github.com/kubernetes/ingress-nginx/blob/3c0bfc1ca3eb48246b12e77d40bde1162633efae/deploy/static/provider/baremetal/deploy.yaml

https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml#L67

If your kubernetes version is 1.22, can you try the beta release of the controller and provide feedback https://github.com/kubernetes/ingress-nginx/releases/download/helm-chart-4.0.0/ingress-nginx-4.0.0.tgz

/remove-kind bug /kind feature