kubernetes: DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
Hi,
Since my upgrade to 1.18 version, I have errors in kube-proxy:
E0326 13:14:23.847364 1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[253 0 0 16 2 69 0 0 221 232 251 54 204 98 3 124]
E0326 13:14:23.847388 1 proxier.go:1192] Failed to sync endpoint for service: [fd00:10:96::a]:53/UDP, err: parseIP Error ip=[253 0 0 16 2 69 0 0 221 232 251 54 204 98 3 124]
E0326 13:14:23.847479 1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[253 0 0 16 2 69 0 0 221 232 251 54 204 98 3 124]
E0326 13:14:23.847501 1 proxier.go:1192] Failed to sync endpoint for service: [fd00:10:96::a]:53/TCP, err: parseIP Error ip=[253 0 0 16 2 69 0 0 221 232 251 54 204 98 3 124]
E0326 13:14:23.847595 1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[253 0 0 16 2 69 0 0 221 232 251 54 204 98 3 124]
E0326 13:14:23.847617 1 proxier.go:1192] Failed to sync endpoint for service: [fd00:10:96::a]:9153/TCP, err: parseIP Error ip=[253 0 0 16 2 69 0 0 221 232 251 54 204 98 3 124]
E0326 13:14:23.847706 1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[253 0 0 16 2 69 0 0 192 187 182 147 174 207 103 7]
E0326 13:14:23.847728 1 proxier.go:1192] Failed to sync endpoint for service: [fd00:10:96::7964]:443/TCP, err: parseIP Error ip=[253 0 0 16 2 69 0 0 192 187 182 147 174 207 103 7]
E0326 13:14:23.847813 1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[253 0 0 16 2 69 0 0 192 187 182 147 174 207 103 41]
E0326 13:14:23.847835 1 proxier.go:1192] Failed to sync endpoint for service: [fd00:10:96::dc23]:80/TCP, err: parseIP Error ip=[253 0 0 16 2 69 0 0 192 187 182 147 174 207 103 41]
E0326 13:14:23.848063 1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[253 221 172 173 0 21 1 42 2 80 86 255 254 177 6 5]
E0326 13:14:23.848085 1 proxier.go:1192] Failed to sync endpoint for service: [fd00:10:96::1]:443/TCP, err: parseIP Error ip=[253 221 172 173 0 21 1 42 2 80 86 255 254 177 6 5]
...
I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors.
Do you have also this issue ?
- OS: RHEL7
- Kubernetes version (use
kubectl version
):
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:50:46Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}```
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 40 (33 by maintainers)
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on
Linux CentOS 3.10.0-693.el7.x86_64
.Accroding to the code, netlink will try to get
d.AddressFamily
attribute, but I find the kernel just does not support it 😦This is the
Destination Attributes
defined in/usr/include/linux/ip_vs.h
(kernel 3.10)No
IPVS_DEST_ATTR_ADDR_FAMILY
attribute is defined!But in new kernel version, the
Destination Attributes
is defined like this:Obviously, the kernel has added some attributes (IPVS_DEST_ATTR_ADDR_FAMILY, IPVS_DEST_ATTR_STATS64…).
That is why kube-proxy works well on systems with a higher version of the kernel.
So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
FYI https://github.com/kubernetes/kubernetes/pull/90555
/retitle parseIP error with IPVS proxy
@uablrek Maybe something wrong around here https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/docker/libnetwork/ipvs/netlink.go#L463 Before this commit, it looks like this:
Now it looks like this:
However
d.AddressFamily
is not initialized,it is zero 😦.Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I’ve mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
ipv6-only is beta since 1.18 😄
Made a gist trying to reproduce this in my environment but didn’t got this error. I may try this on some older kernel like from CentOS 7
Added exactly the same IPVS configuration here:
And when running this ‘gist/program’ pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result:
Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
Related to https://github.com/kubernetes/kubernetes/pull/89343/files#diff-71a24186bd4c7e94c34819c55dfa89af?
cc @aojea @thockin