ingress-nginx: Ingress object is binding to wrong IP address

NGINX Ingress controller version: 0.30.0

Kubernetes version (use kubectl version): v1.18.2

Environment:

• Cloud provider or hardware configuration: Intel NUC with dual core 4th gen Intel
• OS (e.g. from /etc/os-release): PRETTY_NAME=“CentOS Linux 7 (Core)”
• Kernel (e.g. uname -a): Linux k8smaster 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kubeadm and Helm
• Others:

What happened:

I am using prometheus-operator in the prometheus namespace. I have a sample nginx web server running for testing purposes in the default namespace where the nginx-ingress controller also resides. nginx-ingress is sitting behind metallb which is tied to IP address 192.168.15.40. However, the prometheus ingress objects appear to be bound to my controller’s IP and not the IP address I’d expect, which is .40. A secondary and possibly related problem, is cert-manager is setting the cert to be valid at ingress.local for the Prometheus related objects and not the wildcard certificate which is bound to the ingress object in the default namespace, where the certificate works.

<details><summary>[kbreit@k8smaster ~]$ kubectl -n prometheus get pod -o wide
NAME                                                     READY   STATUS    RESTARTS   AGE   IP              NODE        NOMINATED NODE   READINESS GATES
alertmanager-prometheus-prometheus-oper-alertmanager-0   2/2     Running   2          39h   172.30.16.167   k8smaster   <none>           <none>
prometheus-grafana-5b47c8cfb6-f8wkl                      2/2     Running   2          39h   172.30.16.162   k8smaster   <none>           <none>
prometheus-kube-state-metrics-6d6fc7946-lllv8            1/1     Running   1          39h   172.30.16.165   k8smaster   <none>           <none>
prometheus-prometheus-node-exporter-mm94x                1/1     Running   1          39h   192.168.15.31   k8smaster   <none>           <none>
prometheus-prometheus-oper-operator-f8df9fcf9-ksct2      2/2     Running   2          39h   172.30.16.169   k8smaster   <none>           <none>
prometheus-prometheus-prometheus-oper-prometheus-0       3/3     Running   1          71m   172.30.16.186   k8smaster   <none>           <none>
[kbreit@k8smaster ~]$ kubectl -n prometheus get ing
NAME                                    CLASS    HOSTS                                ADDRESS         PORTS     AGE
prometheus-grafana                      <none>   grafana-k8s.home.kevinbreit.net      192.168.15.31   80, 443   38h
prometheus-prometheus-oper-prometheus   <none>   prometheus-k8s.home.kevinbreit.net   192.168.15.31   80, 443   39h
[kbreit@k8smaster ~]$ kubectl get pod,ing
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-controller-857967b4f-ktzx2         1/1     Running   0          13h
pod/nginx-ingress-default-backend-7c868597f4-bhkdl   1/1     Running   1          2d14h
pod/nginx-test                                       1/1     Running   0          13h

NAME                              CLASS    HOSTS                            ADDRESS         PORTS   AGE
ingress.extensions/test-ingress   <none>   nginx-test.home.kevinbreit.net   192.168.15.31   80      8d
[kbreit@k8smaster ~]$ kubectl get svc --all-namespaces
NAMESPACE      NAME                                                 TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                        AGE
cert-manager   cert-manager                                         ClusterIP      10.104.238.122   <none>          9402/TCP                       14h
cert-manager   cert-manager-webhook                                 ClusterIP      10.111.106.121   <none>          443/TCP                        14h
default        kubernetes                                           ClusterIP      10.96.0.1        <none>          443/TCP                        8d
default        nginx-ingress-controller                             LoadBalancer   10.108.195.132   192.168.15.40   80:31310/TCP,443:30905/TCP     2d14h
default        nginx-ingress-default-backend                        ClusterIP      10.100.122.240   <none>          80/TCP                         2d14h
default        nginx-test-svc                                       ClusterIP      10.103.55.183    <none>          80/TCP                         8d
kube-system    kube-dns                                             ClusterIP      10.96.0.10       <none>          53/UDP,53/TCP,9153/TCP         8d
kube-system    metrics-server                                       ClusterIP      10.98.78.86      <none>          443/TCP                        2d22h
kube-system    prometheus-prometheus-oper-coredns                   ClusterIP      None             <none>          9153/TCP                       39h
kube-system    prometheus-prometheus-oper-kube-controller-manager   ClusterIP      None             <none>          10252/TCP                      39h
kube-system    prometheus-prometheus-oper-kube-etcd                 ClusterIP      None             <none>          2381/TCP                       39h
kube-system    prometheus-prometheus-oper-kube-proxy                ClusterIP      None             <none>          10249/TCP                      39h
kube-system    prometheus-prometheus-oper-kube-scheduler            ClusterIP      None             <none>          10251/TCP                      39h
kube-system    prometheus-prometheus-oper-kubelet                   ClusterIP      None             <none>          10250/TCP,10255/TCP,4194/TCP   39h
prometheus     alertmanager-operated                                ClusterIP      None             <none>          9093/TCP,9094/TCP,9094/UDP     39h
prometheus     prometheus-grafana                                   ClusterIP      10.105.245.149   <none>          80/TCP                         39h
prometheus     prometheus-kube-state-metrics                        ClusterIP      10.96.156.190    <none>          8080/TCP                       39h
prometheus     prometheus-operated                                  ClusterIP      None             <none>          9090/TCP                       39h
prometheus     prometheus-prometheus-node-exporter                  ClusterIP      10.102.206.105   <none>          9100/TCP                       39h
prometheus     prometheus-prometheus-oper-alertmanager              ClusterIP      10.96.93.173     <none>          9093/TCP                       39h
prometheus     prometheus-prometheus-oper-operator                  ClusterIP      10.102.232.128   <none>          8080/TCP,443/TCP               39h
prometheus     prometheus-prometheus-oper-prometheus                ClusterIP      10.97.93.4       <none>          9090/TCP                       39h</summary></details>

What you expected to happen:

1. I’d expect the Prometheus objects to bind to 192.168.15.40.
2. The certificates should be valid and not show they’re for ingress.local.

/kind bug