kubernetes: kube-proxy iptables load-balancing probability is not equal

Kubernetes version (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“3”, GitVersion:“v1.3.7”, GitCommit:“a2cba278cba1f6881bb0a7704d9cac6fca6ed435”, GitTreeState:“clean”, BuildDate:“2016-09-12T23:15:30Z”, GoVersion:“go1.6.2”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“3”, GitVersion:“v1.3.7”, GitCommit:“a2cba278cba1f6881bb0a7704d9cac6fca6ed435”, GitTreeState:“clean”, BuildDate:“2016-09-12T23:08:43Z”, GoVersion:“go1.6.2”, Compiler:“gc”, Platform:“linux/amd64”}

Environment:

Cloud provider or hardware configuration: Amazon AWS OS (e.g. from /etc/os-release): CentOS Linux 7 (Core) Kernel (e.g. uname -a): Linux bastion 4.7.5-1.el7.elrepo.x86_64 #1 SMP Sat Sep 24 11:54:29 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Install tools: Custom install Others: What happened:

I’ve created a deployment (nginx) with 5 replicas like so: kubectl run nginx --image=nginx --replicas=5

I’ve exposed a service like so: kubectl expose deployment nginx --port=80 --target-port=80

On a worker node, I see this in iptables for this service: -A KUBE-SVC-H2F4SOSDHAEHZFXQ -m comment --comment “default/nginx:” -m statistic --mode random --probability 0.20000000019 -j KUBE-SEP-W565FQEHFWB3IXKJ -A KUBE-SVC-H2F4SOSDHAEHZFXQ -m comment --comment “default/nginx:” -m statistic --mode random --probability 0.25000000000 -j KUBE-SEP-EZJQVRVACWHUOBBQ -A KUBE-SVC-H2F4SOSDHAEHZFXQ -m comment --comment “default/nginx:” -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-FPX6KRWBZCCVJJV6 -A KUBE-SVC-H2F4SOSDHAEHZFXQ -m comment --comment “default/nginx:” -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-PHUVLZZC77CF4GSN -A KUBE-SVC-H2F4SOSDHAEHZFXQ -m comment --comment “default/nginx:” -j KUBE-SEP-DCNJBZVVUJKLG55E

Those probabilities look really odd.

What you expected to happen:

All probabilities should be 0.2.

How to reproduce it (as minimally and precisely as possible):

See above.

Anything else do we need to know:

Looks like it’s line 953 in pkg/proxy/iptables/proxier.go.

It seems like that there is some problem in kube-proxy iptables rules.

// Now write loadbalancing & DNAT rules.
		n := len(endpointChains)
		for i, endpointChain := range endpointChains {
			// Balancing rules in the per-service chain.
			args := []string{
				"-A", string(svcChain),
				"-m", "comment", "--comment", svcName.String(),
			}
			if i < (n - 1) {
				// Each rule is a probabilistic match.
				args = append(args,
					"-m", "statistic",
					"--mode", "random",
					"--probability", fmt.Sprintf("%0.5f", 1.0/float64(n-i)))
			}

In the code above, I find that the probability is not equal, and the total of the probability is above 1. For example:

I have 5 pod in one server, and I use the command iptables-save to see the chain of the iptables, the result is as below:

-A KUBE-SVC-LI77LBOOMGYET5US -m comment --comment "default/showreadiness:showreadiness" -m statistic --mode random --probability 0.20000000019 -j KUBE-SEP-E4QKA7SLJRFZZ2DD 
-A KUBE-SVC-LI77LBOOMGYET5US -m comment --comment "default/showreadiness:showreadiness" -m statistic --mode random --probability 0.25000000000 -j KUBE-SEP-LZ7EGMG4DRXMY26H 
-A KUBE-SVC-LI77LBOOMGYET5US -m comment --comment "default/showreadiness:showreadiness" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-RKIFTWKKG3OHTTMI 
-A KUBE-SVC-LI77LBOOMGYET5US -m comment --comment "default/showreadiness:showreadiness" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-CGDKBCNM24SZWCMS
-A KUBE-SVC-LI77LBOOMGYET5US -m comment --comment "default/showreadiness:showreadiness" -j KUBE-SEP-RI4SRNQQXWSTGE2Y

I find the probability is 1/5, 1/4, 1/3, 1/2, so when the access request is coming, the choose probability of KUBE-SEP-CGDKBCNM24SZWCMS pod is the latest, and I test I send the request to the server at the same time, and a lot of time this pod can handle 3 request, so if I want to every pod handle one task at the same time is not done.