metallb: externalTrafficPolicy does not preserve IP anymore (L2 mode)

- The bug itself, as detailed as you can.

After upgrading some clusters to k8s 1.15.6, target pod cant see source IP anymore. LB service are configured with externalTrafficPolicy: Local. Traffic effectively flows only to the local pod, but the ClientIP is the node one.

- Version of MetalLB

0.8.3

- Version of Kubernetes

1.15.6-rancher1-2

- Name and version of network addon (e.g. Calico, Weave…)

Flannel + Calico (Canal)

- Whether you’ve configured kube-proxy for iptables or ipvs mode

iptables mode.

I tracked rules for a LB service (public IP is hidden) :

# iptables-save |grep GS7VUV4NFC2NA6UK
:KUBE-FW-GS7VUV4NFC2NA6UK - [0:0]
:KUBE-SVC-GS7VUV4NFC2NA6UK - [0:0]
:KUBE-XLB-GS7VUV4NFC2NA6UK - [0:0]
-A KUBE-FW-GS7VUV4NFC2NA6UK -m comment --comment "traefik/traefik:webs loadbalancer IP" -j KUBE-XLB-GS7VUV4NFC2NA6UK
-A KUBE-FW-GS7VUV4NFC2NA6UK -m comment --comment "traefik/traefik:webs loadbalancer IP" -j KUBE-MARK-DROP
-A KUBE-NODEPORTS -p tcp -m comment --comment "traefik/traefik:webs" -m tcp --dport 31163 -j KUBE-XLB-GS7VUV4NFC2NA6UK
-A KUBE-SERVICES -d 10.43.238.216/32 -p tcp -m comment --comment "traefik/traefik:webs cluster IP" -m tcp --dport 443 -j KUBE-SVC-GS7VUV4NFC2NA6UK
-A KUBE-SERVICES -d xxx.xxx.xxx.xxx/32 -p tcp -m comment --comment "traefik/traefik:webs loadbalancer IP" -m tcp --dport 443 -j KUBE-FW-GS7VUV4NFC2NA6UK
-A KUBE-SVC-GS7VUV4NFC2NA6UK -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-7COCHRO3VKLLNBYW
-A KUBE-SVC-GS7VUV4NFC2NA6UK -j KUBE-SEP-YVVGR5GWIKLB47FM
-A KUBE-XLB-GS7VUV4NFC2NA6UK -m comment --comment "masquerade LOCAL traffic for traefik/traefik:webs LB IP" -m addrtype --src-type LOCAL -j KUBE-MARK-MASQ
-A KUBE-XLB-GS7VUV4NFC2NA6UK -m comment --comment "route LOCAL traffic for traefik/traefik:webs LB IP to service chain" -m addrtype --src-type LOCAL -j KUBE-SVC-GS7VUV4NFC2NA6UK
-A KUBE-XLB-GS7VUV4NFC2NA6UK -s 10.42.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-GS7VUV4NFC2NA6UK
-A KUBE-XLB-GS7VUV4NFC2NA6UK -m comment --comment "Balancing rule 0 for traefik/traefik:webs" -j KUBE-SEP-YVVGR5GWIKLB47FM

# iptables-save |grep YVVGR5GWIKLB47FM
:KUBE-SEP-YVVGR5GWIKLB47FM - [0:0]
-A KUBE-SEP-YVVGR5GWIKLB47FM -s 10.42.2.98/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-YVVGR5GWIKLB47FM -p tcp -m tcp -j DNAT --to-destination 10.42.2.98:443
-A KUBE-SVC-GS7VUV4NFC2NA6UK -j KUBE-SEP-YVVGR5GWIKLB47FM
-A KUBE-XLB-GS7VUV4NFC2NA6UK -m comment --comment "Balancing rule 0 for traefik/traefik:webs" -j KUBE-SEP-YVVGR5GWIKLB47FM

I’m not fluent with kube-proxy/iptables, but couldn’t masquerade rule be the cause ?

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 16 (7 by maintainers)

Most upvoted comments

We got it working 😉 In our Service-Spec we had both loadBalancerIP AND externalIPs set to the loadbalanced-ip. The latter one was a remainder of TungstenFabric-SDN.

So it looked like this

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.20.0
    component: controller
    heritage: Tiller
    release: ngix-ingress-public
  name: ngix-ingress-public-nginx-ingress-controller
  namespace: infra-public
spec:
  clusterIP: 10.127.40.106
  externalIPs:
  - HIDDEN
  externalTrafficPolicy: Local
  healthCheckNodePort: 15217
  loadBalancerIP: HIDDEN-BUT-IDENTICAL-TO-externalIPs
  ....

According to https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/ as soon as you have “Service External IP + port” it masqurades. But with loadbalancerIP PLUS externalTrafficPolicy=local traffic is just accepted.

So we removed the externalIPs-configuration from the service-manifest, and immediately the real-client IP shows up.

@remche check if you also have a similar-configuration

iceman91176 on Feb 5, 2020