kubernetes: `externalTrafficPolicy: Local` doesn't work well with nodeport for internal traffic

What happened:

Sending traffic from a pod (hostNetwork: false) to the nodeport (externalTrafficPolicy: Local) on another node is not working properly.

Below illustrate what happens:

To describe it in words:

• We have client podA running on nodeA, server podC running on nodeC.
• Client podA sends request to the nodePort on nodeB.
• On node B, the request gets DNAT to the actual backend podC, but doesn’t get SNAT.
• Request reaches the server podC, and the response goes directly back to the client podA. It can’t go through because the connection path is asymmetrical.

This seems to affect kube-proxy iptables mode but not sure about the others.

This can probably be fixed (if it’s worth it) by adding iptable rules to SNAT the request that goes through nodePort chain, like what happens on nodePort service that uses externalTrafficPolicy: Cluster. Though the source IP will not be preserverd anymore.

What you expected to happen: Pod should be able to send traffic to a nodeport service that uses local externalTrafficPolicy.

How to reproduce it (as minimally and precisely as possible): Created a similar setup as described above.

Anything else we need to know?: cc @freehan @bowei @thockin @kubernetes/sig-network-bugs

Environment:

• Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.0-alpha.0.1370+bfa587631168ae-dirty", GitCommit:"bfa587631168ae53b21fc5215c8e00a0a36e2e75", GitTreeState:"dirty", BuildDate:"2019-01-03T21:44:05Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others: