kubernetes: Failed to access NodePort if kube-proxy running in ipvs mode
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug
What happened: Failed to access a remote NodePort.
How to reproduce it (as minimally and precisely as possible):
- Create a cluster with three nodes (node A,B,C). Network plugin is calico.
- Create a pod on node C and expose it as a NodePort service.
- Node A tries to access NotePort with ip:port where ip is the ip address of node B. Remote node (node A) -> node ip:port (node B) -> remote pod (node C)
Anything else we need to know?: It seems missing masquerade rule on node B. I can see pod receives packet with source ip of node A.
/sig network /area kube-proxy
Environment:
- Kubernetes version (use
kubectl version): 1.8.0 - Calico version: 2.6
- OS (e.g. from /etc/os-release): CoreOS 1465.8.0
- Kernel (e.g.
uname -a): 4.12.14 - Install tools:
- Others:
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 23 (16 by maintainers)
@fasaxc and I are core developers of project calico. We are in the process of adding ipvs support for the project. I think you may have calico running in IPinIP mode so it does an encapsulations for you.
@song-jiang Seems you are right.
I think both the hairpin mode and cross-node masquerade issue can be solved when the PR https://github.com/kubernetes/kubernetes/pull/54219 gets in. This PR is ready for review and any comments are welcome.
Thanks all!
MAKE SENSE. I am trying ipset now and will let you know if I have some updates.
Just in case it wasn’t clear where this came from, @song-jiang and I are core developers on Calico and we’re working on adding support to Calico for IPVS since currently IPVS bypasses Calico’s security policy.
@m1093782566 Yes, that’s the sort of rule. In Calico, that rule is only enabled when IPIP mode is turned on. Normally, Calico (deliberately) avoids adding SNAT to the datapath but that IPIP rule is a special case workaround. We’d like kube-proxy’s rules to be self contained so that it doesn’t rely on Calico (or flannel) for SNAT. I think that’s already the case on some datapaths but node ports weren’t covered by the existing rules.