kubernetes: DualStack: NodePort failed when opening dual-stack with ipvs

@andrewsykim @Lion-Wei just now, i found this 😃

commit 94b5e0f970258828bf163b5ef076da4e4b0802e0
Author: David Ahern <dsa@cumulusnetworks.com>
Date:   Thu Feb 2 08:52:21 2017 -0800

    net: ipv6: Set protocol to kernel for local routes

    IPv6 stack does not set the protocol for local routes, so those routes show
    up with proto "none":
        $ ip -6 ro ls table local
        local ::1 dev lo proto none metric 0  pref medium
        local 2100:3:: dev lo proto none metric 0  pref medium
        local 2100:3::4 dev lo proto none metric 0  pref medium
        local fe80:: dev lo proto none metric 0  pref medium
        ...

    Set rt6i_protocol to RTPROT_KERNEL for consistency with IPv4. Now routes
    show up with proto "kernel":
        $ ip -6 ro ls table local
        local ::1 dev lo proto kernel metric 0  pref medium
        local 2100:3:: dev lo proto kernel metric 0  pref medium
        local 2100:3::4 dev lo proto kernel metric 0  pref medium
        local fe80:: dev lo proto kernel metric 0  pref medium
        ...

    Signed-off-by: David Ahern <dsa@cumulusnetworks.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>

for completeness and to have more context, I’m copying the original description from #93631

because the protocol type in local table for ipv6 is none but not kernal，which is present in local table for ipv4.:

local ::1 dev lo  proto none  metric 0  pref medium
local 100::1 dev lo  proto none  metric 0  pref medium
local 300::3 dev lo  proto none  metric 0  pref medium

local 10.46.177.110 dev ens3  proto kernel  scope host  src 10.46.177.110
broadcast 10.46.177.255 dev ens3  proto kernel  scope link  src 10.46.177.110
local 10.254.0.1 dev kube-ipvs0  proto kernel  scope host  src 10.254.0.1
the case in function 'RouteListFiltered' in the file of vendor/github.com/vishvananda/netlink/route_linux.go could not work well

case filterMask&RT_FILTER_PROTOCOL != 0 && route.Protocol != filter.Protocol:
				continue
because of the inconsistencies between IPv4 and IPv6kernel.
For example, the 'NodePort' is affected and cannot open the local port

it seems that IPVS use the netlink library to dump the route table to get the local addresses https://github.com/kubernetes/kubernetes/blob/04362870ad9f9812f487ab8f89a40f6bf2daa588/pkg/proxy/ipvs/netlink_linux.go#L148

that, causes that if the filter protocol does not match and it can not find the corresponding IPv6 addresses.

That method is only referenced in 2 places, and it’s only used to return IP addresses per interface https://github.com/kubernetes/kubernetes/blob/04362870ad9f9812f487ab8f89a40f6bf2daa588/pkg/proxy/ipvs/proxier.go#L293 https://github.com/kubernetes/kubernetes/blob/04362870ad9f9812f487ab8f89a40f6bf2daa588/pkg/proxy/ipvs/proxier.go#L306

My suggestion for fixing this (I vaguely remember that we discussed this in another PR and someone already submitted something along this lines), is to replace that method for a function that use the go net library. It simplifies a lot the code (looking the routing table for getting the local addresses is more complex than iterating over the current interfaces ip address ), it’s a very simple fix and removes external dependencies, that seems a win-win 😄

An example that can be adapted is:

// getInterfaceIPs returns an array with all the global addresses
// of the interfaces passed as a parameter
func getInterfaceIPs(ifazName string) string {
	var ips string

	ifaz, err := net.InterfaceByName(ifazName)
	if err != nil {
		panic(err.Error())
	}

	addrs, err := ifaz.Addrs()
	if err != nil {
		panic(err.Error())
	}
	for _, addr := range addrs {
		var ip net.IP
		switch v := addr.(type) {
		case *net.IPAddr:
			ip = v.IP
		case *net.IPNet:
			ip = v.IP
		default:
			continue
		}
		if ip.IsGlobalUnicast() {
			ips = ips + "," + ip.String()
		}

	}
	return strings.Trim(ips, ",")
}

I defer to IPVS maintainers the last decision /assign @andrewsykim

/remove-lifecycle stale

I will fix this by using the go net library as @aojea suggested above.

@andrewsykim @Lion-Wei i think maybe it’s a bug of linux. in linux kernel 4.4 there is a function ‘addrconf_dst_alloc’ which has been repaced by ‘addrconf_f6i_alloc’(https://github.com/torvalds/linux/blob/47ec5303d73ea344e84f46660fff693c57641386/net/ipv6/route.c#L4438)now. below shows difference: 4.4:

master:

and the function ‘rt6_fill_node’ 4.4: master: https://github.com/torvalds/linux/blob/47ec5303d73ea344e84f46660fff693c57641386/net/ipv6/route.c#L5494

the thing is more simple for ipv4: https://github.com/torvalds/linux/blob/9bf9511e3d9f328c03f6f79bfb741c3d18f2f2c0/net/ipv4/fib_frontend.c#L1049 the code has never been changed

i install a high version linux：

Linux centos110 5.6.13-1.el7.elrepo.x86_64 #1 SMP Thu May 14 08:05:24 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

and try again：

local ::1 dev lo proto kernel metric 0 pref medium
local 300::1 dev em1 proto kernel metric 0 pref medium

the proto type is correct now.