kubernetes: error creating ipset xxx, error: exit status 1

/kind bug

What happened: Installed k8s 1.11.2 cluster with Kubespray, and selected ipvs instead of iptables as proxy mode. Results in crash loops for all kube-proxy containers:

kube-proxy-k8s-api01 1/1 Running 72 6h kube-proxy-k8s-api02 1/1 Running 72 6h kube-proxy-k8s-node01 0/1 CrashLoopBackOff 74 6h kube-proxy-k8s-node02 0/1 CrashLoopBackOff 71 6h

Output of logs of proxies:

E0811 03:56:46.572470 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-LOAD-BALANCER-SOURCE-IP hash:ip,port,ip inet 1024 65536 0-65535 Kubernetes service load balancer ip + port + source IP for packet filter purpose} map[] 0xc4209237b0} exist, error: error creating ipset KUBE-LOAD-BALANCER-SOURCE-IP, error: exit status 1 E0811 03:56:51.709465 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-LOAD-BALANCER-SOURCE-IP hash:ip,port,ip inet 1024 65536 0-65535 Kubernetes service load balancer ip + port + source IP for packet filter purpose} map[] 0xc4209237b0} exist, error: error creating ipset KUBE-LOAD-BALANCER-SOURCE-IP, error: exit status 1 E0811 03:56:56.772034 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-LOAD-BALANCER-SOURCE-CIDR hash:ip,port,net inet 1024 65536 0-65535 Kubernetes service load balancer ip + port + source cidr for packet filter purpose} map[] 0xc4209237b0} exist, error: error creating ipset KUBE-LOAD-BALANCER-SOURCE-CIDR, error: exit status 1 E0811 03:57:01.913755 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-EXTERNAL-IP hash:ip,port inet 1024 65536 0-65535 Kubernetes service external ip + port for masquerade and filter purpose} map[] 0xc4209237b0} exist, error: error creating ipset KUBE-EXTERNAL-IP, error: exit status 1 E0811 03:57:06.971781 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-LOOP-BACK hash:ip,port,ip inet 1024 65536 0-65535 Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose} map[] 0xc4209237b0} exist, error: error creating ipset KUBE-LOOP-BACK, error: exit status 1 E0811 03:57:12.072673 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-LOOP-BACK hash:ip,port,ip inet 1024 65536 0-65535 Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose} map[] 0xc4209237b0} exist, error: error creating ipset KUBE-LOOP-BACK, error: exit status 1 E0811 03:57:17.277255 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-NODE-PORT-LOCAL-UDP bitmap:port inet 1024 65536 0-65535 Kubernetes nodeport UDP port with externalTrafficPolicy=local} map[] 0xc4209237b0} exist, error: error creating ipset KUBE-NODE-PORT-LOCAL-UDP, error: exit status 1 E0811 03:57:22.372453 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-NODE-PORT-LOCAL-TCP bitmap:port inet 1024 65536 0-65535 Kubernetes nodeport TCP port with externalTrafficPolicy=local} map[] 0xc4209237b0} exist, error: error creating ipset KUBE-NODE-PORT-LOCAL-TCP, error: exit status 1 E0811 03:57:27.475447 1 ipset.go:156] Failed to make sure ip set: &{{KUBE-EXTERNAL-IP hash:ip,port inet 1024 65536 0-65535 Kubernetes service external ip + port for masquerade and filter purpose} map[] 0xc4209237b0} exist, error: error creating ipset KUBE-EXTERNAL-IP, error: exit status 1

What you expected to happen:

Stable kube-proxy containers.

kube-proxy-k8s-api01 1/1 Running 0 6h kube-proxy-k8s-api02 1/1 Running 0 6h kube-proxy-k8s-node01 1/1 Running 0 6h kube-proxy-k8s-node02 1/1 Running 0 6h

How to reproduce it (as minimally and precisely as possible):

Install kubernetes 1.11.x with kube_proxy_mode: ipvs

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.11.2

• Cloud provider or hardware configuration: bare metal

• OS (e.g. from /etc/os-release): CoreOS stable (1800.6.0)

• Kernel (e.g. uname -a): Linux k8s-acc-api02 4.14.59-coreos-r2 #1 SMP Sat Aug 4 02:49:25 UTC 2018 x86_64 Intel® Xeon® CPU E5-2630L 0 @ 2.00GHz GenuineIntel GNU/Linux

• Install tools: Kubespray

• Others: Probably doesn’t matter, but using 3 separate etcd nodes.