kubernetes: DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7

Hi,

Since my upgrade to 1.18 version, I have errors in kube-proxy:

E0326 13:14:23.847364       1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[253 0 0 16 2 69 0 0 221 232 251 54 204 98 3 124]
E0326 13:14:23.847388       1 proxier.go:1192] Failed to sync endpoint for service: [fd00:10:96::a]:53/UDP, err: parseIP Error ip=[253 0 0 16 2 69 0 0 221 232 251 54 204 98 3 124]
E0326 13:14:23.847479       1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[253 0 0 16 2 69 0 0 221 232 251 54 204 98 3 124]
E0326 13:14:23.847501       1 proxier.go:1192] Failed to sync endpoint for service: [fd00:10:96::a]:53/TCP, err: parseIP Error ip=[253 0 0 16 2 69 0 0 221 232 251 54 204 98 3 124]
E0326 13:14:23.847595       1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[253 0 0 16 2 69 0 0 221 232 251 54 204 98 3 124]
E0326 13:14:23.847617       1 proxier.go:1192] Failed to sync endpoint for service: [fd00:10:96::a]:9153/TCP, err: parseIP Error ip=[253 0 0 16 2 69 0 0 221 232 251 54 204 98 3 124]
E0326 13:14:23.847706       1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[253 0 0 16 2 69 0 0 192 187 182 147 174 207 103 7]
E0326 13:14:23.847728       1 proxier.go:1192] Failed to sync endpoint for service: [fd00:10:96::7964]:443/TCP, err: parseIP Error ip=[253 0 0 16 2 69 0 0 192 187 182 147 174 207 103 7]
E0326 13:14:23.847813       1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[253 0 0 16 2 69 0 0 192 187 182 147 174 207 103 41]
E0326 13:14:23.847835       1 proxier.go:1192] Failed to sync endpoint for service: [fd00:10:96::dc23]:80/TCP, err: parseIP Error ip=[253 0 0 16 2 69 0 0 192 187 182 147 174 207 103 41]
E0326 13:14:23.848063       1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[253 221 172 173 0 21 1 42 2 80 86 255 254 177 6 5]
E0326 13:14:23.848085       1 proxier.go:1192] Failed to sync endpoint for service: [fd00:10:96::1]:443/TCP, err: parseIP Error ip=[253 221 172 173 0 21 1 42 2 80 86 255 254 177 6 5]
...

I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors.

Do you have also this issue ?

  • OS: RHEL7
  • Kubernetes version (use kubectl version):
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:50:46Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}```

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 40 (33 by maintainers)

Most upvoted comments

After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on Linux CentOS 3.10.0-693.el7.x86_64.

Accroding to the code, netlink will try to get d.AddressFamily attribute, but I find the kernel just does not support it 😦

This is the Destination Attributes defined in /usr/include/linux/ip_vs.h (kernel 3.10)

/*
 * Attributes used to describe a destination (real server)
 *
 * Used inside nested attribute IPVS_CMD_ATTR_DEST
 */
enum {
	IPVS_DEST_ATTR_UNSPEC = 0,
	IPVS_DEST_ATTR_ADDR,		/* real server address */
	IPVS_DEST_ATTR_PORT,		/* real server port */

	IPVS_DEST_ATTR_FWD_METHOD,	/* forwarding method */
	IPVS_DEST_ATTR_WEIGHT,		/* destination weight */

	IPVS_DEST_ATTR_U_THRESH,	/* upper threshold */
	IPVS_DEST_ATTR_L_THRESH,	/* lower threshold */

	IPVS_DEST_ATTR_ACTIVE_CONNS,	/* active connections */
	IPVS_DEST_ATTR_INACT_CONNS,	/* inactive connections */
	IPVS_DEST_ATTR_PERSIST_CONNS,	/* persistent connections */

	IPVS_DEST_ATTR_STATS,		/* nested attribute for dest stats */
	__IPVS_DEST_ATTR_MAX,
};

No IPVS_DEST_ATTR_ADDR_FAMILY attribute is defined!

But in new kernel version, the Destination Attributes is defined like this:

/*
 * Attributes used to describe a destination (real server)
 *
 * Used inside nested attribute IPVS_CMD_ATTR_DEST
 */
enum {
	IPVS_DEST_ATTR_UNSPEC = 0,
	IPVS_DEST_ATTR_ADDR,		/* real server address */
	IPVS_DEST_ATTR_PORT,		/* real server port */

	IPVS_DEST_ATTR_FWD_METHOD,	/* forwarding method */
	IPVS_DEST_ATTR_WEIGHT,		/* destination weight */

	IPVS_DEST_ATTR_U_THRESH,	/* upper threshold */
	IPVS_DEST_ATTR_L_THRESH,	/* lower threshold */

	IPVS_DEST_ATTR_ACTIVE_CONNS,	/* active connections */
	IPVS_DEST_ATTR_INACT_CONNS,	/* inactive connections */
	IPVS_DEST_ATTR_PERSIST_CONNS,	/* persistent connections */

	IPVS_DEST_ATTR_STATS,		/* nested attribute for dest stats */

	IPVS_DEST_ATTR_ADDR_FAMILY,	/* Address family of address */

	IPVS_DEST_ATTR_STATS64,		/* nested attribute for dest stats */

	IPVS_DEST_ATTR_TUN_TYPE,	/* tunnel type */

	IPVS_DEST_ATTR_TUN_PORT,	/* tunnel port */

	IPVS_DEST_ATTR_TUN_FLAGS,	/* tunnel flags */

	__IPVS_DEST_ATTR_MAX,
};

Obviously, the kernel has added some attributes (IPVS_DEST_ATTR_ADDR_FAMILY, IPVS_DEST_ATTR_STATS64…).

That is why kube-proxy works well on systems with a higher version of the kernel.

So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.

/retitle parseIP error with IPVS proxy

@uablrek Maybe something wrong around here https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/docker/libnetwork/ipvs/netlink.go#L463 Before this commit, it looks like this:

ip, err := parseIP(attr.Value, syscall.AF_INET)

Now it looks like this:

ip, err := parseIP(attr.Value, d.AddressFamily)

However d.AddressFamily is not initialized,it is zero 😦.

Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I’ve mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now

However ipv6-only and dual-stack are in “alpha”

ipv6-only is beta since 1.18 😄

Made a gist trying to reproduce this in my environment but didn’t got this error. I may try this on some older kernel like from CentOS 7

Added exactly the same IPVS configuration here:

ipvsadm -A -t [fd00:10:96::1]:443 -s lc
ipvsadm -a -t [fd00:10:96::1]:443 -r [fddd:acad:15:12a:250:56ff:feb1:605]:6443 -m
ipvsadm -L -n
ipvsadm -A -t [fd00:10:96::a]:53 -s lc
ipvsadm -L -n
ipvsadm -a -t [fd00:10:96::a]:53 -r [fd00:10:245:0:c0bb:b693:aecf:6727]:53
ipvsadm -a -t [fd00:10:96::a]:53 -r [fd00:10:245:0:dde8:fb36:cc62:37c]:53
ipvsadm -A -u [fd00:10:96::a]:53 -s lc
ipvsadm -a -u [fd00:10:96::a]:53 -r [fd00:10:245:0:dde8:fb36:cc62:37c]:53
ipvsadm -a -u [fd00:10:96::a]:53 -r [fd00:10:245:0:c0bb:b693:aecf:6727]:53

And when running this ‘gist/program’ pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result:

&{fd00:10:245:0:c0bb:b693:aecf:6727 53 1 0 0}&{fd00:10:245:0:dde8:fb36:cc62:37c 53 1 0 0}

Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.