kubernetes: IPset missing from gcr.io/google_containers/hyperkube-amd64

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened: Upgraded to Kuberenetes 1.9 and now IPVS mode in kube-proxy stopped working due to

I1218 09:59:28.446570       1 feature_gate.go:184] feature gates: map[SupportIPVSProxyMode:true]
E1218 09:59:28.463326       1 server_others.go:299] can't determine whether to use ipvs proxy, error: error getting ipset version, error: executable file not found in $PATH
I1218 09:59:28.464584       1 server_others.go:138] Using iptables Proxier

What you expected to happen:

Normal startup as it’s been working flawless in 1.8.5

How to reproduce it (as minimally and precisely as possible): Use kube-proxy with IPVS service mode on 1.9.0 hyperkube image from GCR

Anything else we need to know?: If i exec into the running kube-proxy pod and do apt install ipset then restart the container kube-proxy starts in ipvs mode fine

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: On-prem
  • OS (e.g. from /etc/os-release): CoreOS Container Linux by CoreOS stable (1576.4.0)
  • Kernel (e.g. uname -a): 4.13.16-coreos-r2
  • Install tools: Own
  • Others:
docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.4
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:24:58 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.4
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:24:58 2017
 OS/Arch:      linux/amd64
 Experimental: false

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 15 (6 by maintainers)

Commits related to this issue

Most upvoted comments

@roffe @dixudx @ivanilves So kube-proxy-amd64:v1.9.0 does have ipset. So please use that. However hyperkube-amd64:v1.9.0 does not have ipset (see below).

The commit above https://github.com/kubernetes/kubernetes/commit/787a55bb67ccd2da14aa6e7f91289c859beecb5f was merged well before we shipped v1.9.0, But hyperkube image does not use the debian-iptables, it uses debian-hyperkube-base image if you see https://github.com/kubernetes/kubernetes/blob/master/cluster/images/hyperkube/Makefile#L24

Anyone want to log a PR for hyperkube image to have ipset?

[dims@bigbox 07:49] ~ ⟩ docker run -it gcr.io/google_containers/kube-proxy-amd64:v1.9.0
Unable to find image 'gcr.io/google_containers/kube-proxy-amd64:v1.9.0' locally
v1.9.0: Pulling from google_containers/kube-proxy-amd64
d0c199d1a37e: Already exists
743a743e06dd: Already exists
867cb82bc63d: Pull complete
Digest: sha256:3760355108b97ed0ff74d33fbec72764e1d7f8fbdb06ed0934117196819e3d00
Status: Downloaded newer image for gcr.io/google_containers/kube-proxy-amd64:v1.9.0
# which ipset
/sbin/ipset
# exit
[dims@bigbox 07:52] ~ ⟩ docker run -it gcr.io/google_containers/hyperkube-amd64:v1.9.0
sh-4.4# which ipset
sh-4.4# exit
exit

FYI: I’ve rebuilt hyperkube image for CoreOS https://hub.docker.com/r/ivanilves/hyperkube/ with ipset installed.

Just for testing purposes. Everything works for me with it. Waiting for the next release. ☺️

Can we re-open this, #57648 only fixes this on debian, not amd64!

hyperkube image is based on https://github.com/kubernetes/kubernetes/tree/master/build/debian-hyperkube-base (which is based on https://github.com/kubernetes/kubernetes/tree/master/build/debian-base ) kube-proxy (and flannel) has a separate image based on https://github.com/kubernetes/kubernetes/tree/master/build/debian-iptables But as hyperkube has included kube-proxy support, IMO also debian-hyperkube-base should be extended by ipset command, after this https://github.com/kubernetes/kubernetes/blob/e378a2a6c3a51b2b016165e0c10a3dd01ce99cb0/build/debian-hyperkube-base/Dockerfile#L35 line.