kubernetes: kube-proxy:1.17 for s390x image is broken?

What happened: When I tried to use kubernetes 1.17 for s390x with kubeadm, I got below messages in the kube-proxy pod. It ended up failure of installing cni like flannel. But if I switch back to kubernetes 1.15, everything works fine without these messages. Checked it with sig-network in the k8s slack, there could be something broken in the image kube-proxy:1.17 for s390x.

E0108 13:16:53.070052       1 proxier.go:787] Failed to ensure that filter chain KUBE-EXTERNAL-SERVICES exists: error creating chain "KUBE-EXTERNAL-SERVICES": exit status 2: update-alternatives: error: alternative /usr/sbin/iptables-legacy for iptables not registered; not setting
I0108 13:16:53.070129       1 proxier.go:779] Sync failed; retrying in 30s
W0108 13:17:22.902894       1 iptables.go:508] Could not set up iptables canary mangle/KUBE-PROXY-CANARY: error creating chain "KUBE-PROXY-CANARY": exit status 2: update-alternatives: error: alternative /usr/sbin/iptables-legacy for iptables not registered; not setting

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version): 1.17
Cloud provider or hardware configuration: s390x
OS (e.g: cat /etc/os-release): rhel7.7
Kernel (e.g. uname -a): 3.10.0-1062.9.1.el7.s390x #1 SMP Mon Dec 2 08:33:00 EST 2019 s390x s390x s390x GNU/Linux
Install tools: kubeadm
Network plugin and version (if this is a network-related bug): flannel
Others: image name k8s.gcr.io/kube-proxy:v1.17.0

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 73 (41 by maintainers)

Most upvoted comments

Do you know about issues with cross-compiled images on s390x?

The problem doesn’t really appear to be cross-compiling…

Well… I guess if you were doing really hacky stuff somehow then there might be problems. The error is consistent with the possibility that the s390x image has all of the right packages installed but the post-install scripts didn’t get run when installing them…

danwinship on Feb 10, 2020

@prankkelkar - that’s essentially what our build process does: https://github.com/kubernetes/kubernetes/blob/master/build/lib/release.sh#L391-L394

tallclair on Mar 17, 2020

@BenTheElder I debugged further on the base images used. Looks like current version of qemu-user-static does not work with debian:buster base image while cross compiling for s390x. I was able to cross compile kube proxy image and verify it successfully by upgrading qemu-user-static to latest stable version i.e v4.2.0-6. It would be nice if we can upgrade QEMUVERSION and fix this issue.

prankkelkar on Mar 16, 2020

I tried cross compiling debian-iptables-s390x:v12.0.1 image using steps from comment. Observed following error in logs but it did not cause dockerfile build to fail.

Step 1/6 : FROM k8s.gcr.io/debian-base-s390x:v2.0.0
v2.0.0: Pulling from debian-base-s390x
29aaea6da3fd: Pull complete
Digest: sha256:78ef2a6b017539379c1654b4e52ba8519bfec821c62d0b3a1dbd15104b711e21
Status: Downloaded newer image for k8s.gcr.io/debian-base-s390x:v2.0.0
 ---> fcbbb7cb47b0
Step 2/6 : RUN echo deb http://deb.debian.org/debian buster-backports main >> /etc/apt/sources.list;     apt-get update;     apt-get -t buster-backports -y --no-install-recommends install iptables
 ---> Running in d6c098077a56
Inconsistency detected by ld.so: rtld.c: 721: init_tls: Assertion `i == GL(dl_tls_max_dtv_idx)' failed!
Inconsistency detected by ld.so: rtld.c: 721: init_tls: Assertion `i == GL(dl_tls_max_dtv_idx)' failed!
Removing intermediate container d6c098077a56
 ---> 4f8ba854ee9c
Step 3/6 : RUN clean-install     conntrack     ebtables     ipset     kmod     netbase
 ---> Running in 4cf66989acfa
Inconsistency detected by ld.so: rtld.c: 721: init_tls: Assertion `i == GL(dl_tls_max_dtv_idx)' failed!
Inconsistency detected by ld.so: rtld.c: 721: init_tls: Assertion `i == GL(dl_tls_max_dtv_idx)' failed!
Inconsistency detected by ld.so: rtld.c: 721: init_tls: Assertion `i == GL(dl_tls_max_dtv_idx)' failed!
/bin/rm: cannot remove '': Directory not empty
Removing intermediate container 4cf66989acfa
 ---> 8544e901b686
Step 4/6 : COPY iptables-wrapper /usr/sbin/iptables-wrapper
 ---> 85b88e87904f
Step 5/6 : RUN update-alternatives      --install /usr/sbin/iptables iptables /usr/sbin/iptables-wrapper 100    --slave /usr/sbin/iptables-restore iptables-restore /usr/sbin/iptables-wrapper        --slave /usr/sbin/iptables-save iptables-save /usr/sbin/iptables-wrapper
 ---> Running in fc4dbce488b4
update-alternatives: using /usr/sbin/iptables-wrapper to provide /usr/sbin/iptables (iptables) in auto mode
Removing intermediate container fc4dbce488b4
 ---> 6cf573949b48
Step 6/6 : RUN update-alternatives      --install /usr/sbin/ip6tables ip6tables /usr/sbin/iptables-wrapper 100  --slave /usr/sbin/ip6tables-restore ip6tables-restore /usr/sbin/iptables-wrapper      --slave /usr/sbin/ip6tables-save ip6tables-save /usr/sbin/iptables-wrapper
 ---> Running in 29d591f69e92
update-alternatives: using /usr/sbin/iptables-wrapper to provide /usr/sbin/ip6tables (ip6tables) in auto mode
Removing intermediate container 29d591f69e92
 ---> 685f25d31efb
Successfully built 685f25d31efb
Successfully tagged staging-k8s.gcr.io/debian-iptables-s390x:v12.0.1

After inspecting Base image for above dockerfile i.e k8s.gcr.io/debian-base-s390x:v2.0.0 found following outputs on mentioned architectures. amd64:

root@Intel-pk# docker run -it --rm k8s.gcr.io/debian-base-s390x:v2.0.0
# echo deb http://deb.debian.org/debian buster-backports main >> /etc/apt/sources.list
# apt-get update
Inconsistency detected by ld.so: rtld.c: 721: init_tls: Assertion `i == GL(dl_tls_max_dtv_idx)' failed!

s390x

pk@pras1:~$ docker run -it --rm k8s.gcr.io/debian-base-s390x:v2.0.0
# echo deb http://deb.debian.org/debian buster-backports main >> /etc/apt/sources.list
# apt-get update
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:1 http://security-cdn.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]
Get:4 http://deb.debian.org/debian buster-backports InRelease [46.7 kB]
Get:5 http://deb.debian.org/debian buster/main s390x Packages [7624 kB]
Get:6 http://security-cdn.debian.org/debian-security buster/updates/main s390x Packages [161 kB]
Get:7 http://deb.debian.org/debian buster-updates/main s390x Packages [7380 B]
Get:8 http://deb.debian.org/debian buster-backports/main s390x Packages [221 kB]
Fetched 8296 kB in 5s (1795 kB/s)
Reading package lists... Done

We need check what exactly is wrong with debian-base-s390x image.

prankkelkar on Mar 13, 2020

Use follow steps to get a v1.17.2 cluster with none driver on s390x machine by minikube

Build debian-iptables on the s390x machine

git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes/build/debian-iptables
make build ARCH=s390x
...
Successfully built 89353f449fa3
Successfully tagged staging-k8s.gcr.io/debian-iptables-s390x:v12.0.1

Get kube-proxy binary file from k8s.gcr.io/kube-proxy:v1.17.2

cd /root
docker run --rm -v /root:/tmp k8s.gcr.io/kube-proxy:v1.17.2 cp /usr/local/bin/kube-proxy /tmp

Build a workable kube-proxy:v1.17.2 image

cat <<EOF > "Dockerfile"
FROM staging-k8s.gcr.io/debian-iptables-s390x:v12.0.1
COPY kube-proxy /usr/local/bin/kube-proxy
EOF
docker build . -t k8s.gcr.io/kube-proxy:v1.17.2

Test the built out k8s.gcr.io/kube-proxy:v1.17.2 image

docker run k8s.gcr.io/kube-proxy:v1.17.2 iptables   
...
iptables v1.8.3 (legacy): no command specified
Try `iptables -h' or 'iptables --help' for more information.

Use minikube create v1.17.2 cluster

minikube start --vm-driver=none --kubernetes-version=1.17.2

Check cluster status

root@c21091c210d5:~/minikube/out# kubectl get po -A
NAMESPACE     NAME                                   READY   STATUS         RESTARTS   AGE
kube-system   coredns-6955765f44-9pkvs               1/1     Running        0          31s
kube-system   coredns-6955765f44-tlt9g               1/1     Running        0          31s
kube-system   etcd-c21091c210d5                      1/1     Running        0          18s
kube-system   kube-apiserver-c21091c210d5            1/1     Running        0          18s
kube-system   kube-controller-manager-c21091c210d5   1/1     Running        0          18s
kube-system   kube-proxy-kmkd6                       1/1     Running        0          31s
kube-system   kube-scheduler-c21091c210d5            1/1     Running        0          18s
kube-system   storage-provisioner                    0/1     ErrImagePull   0          37s
root@c21091c210d5:~/minikube/out# kubectl logs kube-proxy-kmkd6 -n kube-system
W0220 10:09:42.744674       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
I0220 10:09:42.755135       1 node.go:135] Successfully retrieved node IP: 10.23.2.69
I0220 10:09:42.755171       1 server_others.go:145] Using iptables Proxier.
W0220 10:09:42.755255       1 proxier.go:286] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0220 10:09:42.755414       1 server.go:571] Version: v1.17.2
I0220 10:09:42.755951       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0220 10:09:42.756669       1 config.go:313] Starting service config controller
I0220 10:09:42.756768       1 shared_informer.go:197] Waiting for caches to sync for service config
I0220 10:09:42.756833       1 config.go:131] Starting endpoints config controller
I0220 10:09:42.756857       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
I0220 10:09:42.858192       1 shared_informer.go:204] Caches are synced for service config 
I0220 10:09:42.858266       1 shared_informer.go:204] Caches are synced for endpoints config 
root@c21091c210d5:~/minikube/out#

liudalibj on Feb 20, 2020

kube-proxy release images are built by the release team…? I built the debian base images using the make rules but not kube-proxy.

BenTheElder on Feb 8, 2020