kubernetes: IPVS is not working with hostport

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version):

Environment:

• Kubernetes version (use kubectl version): 1.11.0
• Cloud provider or hardware configuration: None
• OS (e.g. from /etc/os-release): Ubuntu 16.04
• Kernel (e.g. uname -a):
• Others:

What happened?

I’m trying to set up a kubernetes (v1.11.0) cluster. Here is the setting of the cluster:

1. Enable IPVS in kube-proxy
2. Use hostport for nginx ingress controller. Deloy nginx ingress controller on the same node as kubenetes core services.

Host port 443 and 80 of nginx ingress controller is conflict with kubernetes (kube-apiserver) service’s port 443 and 80. Nginx ingress controller keeps crashloopback because failed to connect to kube-apiserver. I even cannot run telnet 10.0.0.1 443.

Try to find the root cause. Here is the iptable rules generated

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
cali-PREROUTING  all  --  anywhere             anywhere             /* cali:6gwbT8clXdHdC1b1 */
CNI-HOSTPORT-DNAT  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
cali-OUTPUT  all  --  anywhere             anywhere             /* cali:tVnHkvAo15HuiPy0 */
CNI-HOSTPORT-DNAT  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
cali-POSTROUTING  all  --  anywhere             anywhere             /* cali:O3lYWMrLQYEMJtB5 */
CNI-HOSTPORT-SNAT  all  --  localhost           !localhost
KUBE-POSTROUTING  all  --  anywhere             anywhere             /* kubernetes postrouting rules */
MASQUERADE  all  --  172.17.0.0/16        anywhere

The CNI-HOSTPORT related iptable rules are on top of KUBE-SERVICES rule. As you seen the pre-routing rule

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
cali-PREROUTING  all  --  anywhere             anywhere             /* cali:6gwbT8clXdHdC1b1 */
CNI-HOSTPORT-DNAT  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Every time I call https://10.0.0.1:443 it will match CNI-HOSTPORT-DNAT rule to redirect to nginx ingress service. That’s why I got failure when calling kubernetes api with service ip.

What you expected to happen?

Should always take KUBE-SERVICES rule first rather than CNI-HOSTPORT-DNAT. So the pre-routing rule should be like

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
cali-PREROUTING  all  --  anywhere             anywhere             /* cali:6gwbT8clXdHdC1b1 */
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
CNI-HOSTPORT-DNAT  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

How to reproduce it (as minimally and precisely as possible)?

See above append.

Anything else we need to know?

When I restarted kube-proxy, the iptable rules were generated correctly. Nginx ingress controller now started successfully.