kubernetes: DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7

After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on Linux CentOS 3.10.0-693.el7.x86_64.

Accroding to the code, netlink will try to get d.AddressFamily attribute, but I find the kernel just does not support it 😦

This is the Destination Attributes defined in /usr/include/linux/ip_vs.h (kernel 3.10)

/*
 * Attributes used to describe a destination (real server)
 *
 * Used inside nested attribute IPVS_CMD_ATTR_DEST
 */
enum {
	IPVS_DEST_ATTR_UNSPEC = 0,
	IPVS_DEST_ATTR_ADDR,		/* real server address */
	IPVS_DEST_ATTR_PORT,		/* real server port */

	IPVS_DEST_ATTR_FWD_METHOD,	/* forwarding method */
	IPVS_DEST_ATTR_WEIGHT,		/* destination weight */

	IPVS_DEST_ATTR_U_THRESH,	/* upper threshold */
	IPVS_DEST_ATTR_L_THRESH,	/* lower threshold */

	IPVS_DEST_ATTR_ACTIVE_CONNS,	/* active connections */
	IPVS_DEST_ATTR_INACT_CONNS,	/* inactive connections */
	IPVS_DEST_ATTR_PERSIST_CONNS,	/* persistent connections */

	IPVS_DEST_ATTR_STATS,		/* nested attribute for dest stats */
	__IPVS_DEST_ATTR_MAX,
};

No IPVS_DEST_ATTR_ADDR_FAMILY attribute is defined!

But in new kernel version, the Destination Attributes is defined like this:

/*
 * Attributes used to describe a destination (real server)
 *
 * Used inside nested attribute IPVS_CMD_ATTR_DEST
 */
enum {
	IPVS_DEST_ATTR_UNSPEC = 0,
	IPVS_DEST_ATTR_ADDR,		/* real server address */
	IPVS_DEST_ATTR_PORT,		/* real server port */

	IPVS_DEST_ATTR_FWD_METHOD,	/* forwarding method */
	IPVS_DEST_ATTR_WEIGHT,		/* destination weight */

	IPVS_DEST_ATTR_U_THRESH,	/* upper threshold */
	IPVS_DEST_ATTR_L_THRESH,	/* lower threshold */

	IPVS_DEST_ATTR_ACTIVE_CONNS,	/* active connections */
	IPVS_DEST_ATTR_INACT_CONNS,	/* inactive connections */
	IPVS_DEST_ATTR_PERSIST_CONNS,	/* persistent connections */

	IPVS_DEST_ATTR_STATS,		/* nested attribute for dest stats */

	IPVS_DEST_ATTR_ADDR_FAMILY,	/* Address family of address */

	IPVS_DEST_ATTR_STATS64,		/* nested attribute for dest stats */

	IPVS_DEST_ATTR_TUN_TYPE,	/* tunnel type */

	IPVS_DEST_ATTR_TUN_PORT,	/* tunnel port */

	IPVS_DEST_ATTR_TUN_FLAGS,	/* tunnel flags */

	__IPVS_DEST_ATTR_MAX,
};

Obviously, the kernel has added some attributes (IPVS_DEST_ATTR_ADDR_FAMILY, IPVS_DEST_ATTR_STATS64…).

That is why kube-proxy works well on systems with a higher version of the kernel.

So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.

FYI https://github.com/kubernetes/kubernetes/pull/90555

/retitle parseIP error with IPVS proxy

@uablrek Maybe something wrong around here https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/docker/libnetwork/ipvs/netlink.go#L463 Before this commit, it looks like this:

ip, err := parseIP(attr.Value, syscall.AF_INET)

Now it looks like this:

ip, err := parseIP(attr.Value, d.AddressFamily)

However d.AddressFamily is not initialized，it is zero 😦.

Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I’ve mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now

However ipv6-only and dual-stack are in “alpha”

ipv6-only is beta since 1.18 😄

Made a gist trying to reproduce this in my environment but didn’t got this error. I may try this on some older kernel like from CentOS 7

Added exactly the same IPVS configuration here:

ipvsadm -A -t [fd00:10:96::1]:443 -s lc
ipvsadm -a -t [fd00:10:96::1]:443 -r [fddd:acad:15:12a:250:56ff:feb1:605]:6443 -m
ipvsadm -L -n
ipvsadm -A -t [fd00:10:96::a]:53 -s lc
ipvsadm -L -n
ipvsadm -a -t [fd00:10:96::a]:53 -r [fd00:10:245:0:c0bb:b693:aecf:6727]:53
ipvsadm -a -t [fd00:10:96::a]:53 -r [fd00:10:245:0:dde8:fb36:cc62:37c]:53
ipvsadm -A -u [fd00:10:96::a]:53 -s lc
ipvsadm -a -u [fd00:10:96::a]:53 -r [fd00:10:245:0:dde8:fb36:cc62:37c]:53
ipvsadm -a -u [fd00:10:96::a]:53 -r [fd00:10:245:0:c0bb:b693:aecf:6727]:53

And when running this ‘gist/program’ pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result:

&{fd00:10:245:0:c0bb:b693:aecf:6727 53 1 0 0}&{fd00:10:245:0:dde8:fb36:cc62:37c 53 1 0 0}

Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.

Related to https://github.com/kubernetes/kubernetes/pull/89343/files#diff-71a24186bd4c7e94c34819c55dfa89af?

cc @aojea @thockin