kubernetes: kubeadm and kubelet 1.15 fail to install on centos 7 after patches released today

What happened:

Attempted to install kubernetes 1.15 binaries following documentation on Centos 7 (https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/) and received failures:

# yum install kubelet-1.15* kubeadm-1.15* kubectl-1.15*
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: repos.dfw.quadranet.com
 * extras: linux-mirrors.fnal.gov
 * updates: mirrors.raystedman.org
Resolving Dependencies
--> Running transaction check
---> Package kubeadm.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
--> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.15.12-0.x86_64
---> Package kubectl.x86_64 0:1.15.12-0 will be installed
---> Package kubelet.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubelet-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
--> Processing Dependency: iptables >= 1.4.21 for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: socat for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: iproute for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: ethtool for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: ebtables for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: conntrack for package: kubelet-1.15.12-0.x86_64
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-7.el7 will be installed
--> Processing Dependency: libnetfilter_conntrack >= 1.0.6 for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libmnl.so.0(LIBMNL_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnfnetlink.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_conntrack.so.3()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libmnl.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
---> Package cri-tools.x86_64 0:1.13.0-0 will be installed
---> Package ebtables.x86_64 0:2.0.10-16.el7 will be installed
---> Package ethtool.x86_64 2:4.8-10.el7 will be installed
---> Package iproute.x86_64 0:4.11.0-25.el7_7.2 will be installed
---> Package iptables.x86_64 0:1.4.21-34.el7 will be installed
---> Package kubeadm.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
---> Package kubelet.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubelet-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed
--> Processing Dependency: libwrap.so.0()(64bit) for package: socat-1.7.3.2-2.el7.x86_64
--> Running transaction check
---> Package kubeadm.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
---> Package kubelet.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubelet-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
---> Package libmnl.x86_64 0:1.0.3-7.el7 will be installed
---> Package libnetfilter_conntrack.x86_64 0:1.0.6-1.el7_3 will be installed
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
---> Package libnfnetlink.x86_64 0:1.0.1-4.el7 will be installed
---> Package tcp_wrappers-libs.x86_64 0:7.6-77.el7 will be installed
--> Finished Dependency Resolution
Error: Package: kubelet-1.15.12-0.x86_64 (kubernetes)
           Requires: kubernetes-cni >= 0.7.5
           Available: kubernetes-cni-0.3.0.1-0.07a8a2.x86_64 (kubernetes)
               kubernetes-cni = 0.3.0.1-0.07a8a2
           Available: kubernetes-cni-0.5.1-0.x86_64 (kubernetes)
               kubernetes-cni = 0.5.1-0
           Available: kubernetes-cni-0.5.1-1.x86_64 (kubernetes)
               kubernetes-cni = 0.5.1-1
           Available: kubernetes-cni-0.6.0-0.x86_64 (kubernetes)
               kubernetes-cni = 0.6.0-0
           Available: kubernetes-cni-0.7.5-0.x86_64 (kubernetes)
               kubernetes-cni = 0.7.5-0
Error: Package: kubeadm-1.15.12-0.x86_64 (kubernetes)
           Requires: kubernetes-cni >= 0.7.5
           Available: kubernetes-cni-0.3.0.1-0.07a8a2.x86_64 (kubernetes)
               kubernetes-cni = 0.3.0.1-0.07a8a2
           Available: kubernetes-cni-0.5.1-0.x86_64 (kubernetes)
               kubernetes-cni = 0.5.1-0
           Available: kubernetes-cni-0.5.1-1.x86_64 (kubernetes)
               kubernetes-cni = 0.5.1-1
           Available: kubernetes-cni-0.6.0-0.x86_64 (kubernetes)
               kubernetes-cni = 0.6.0-0
           Available: kubernetes-cni-0.7.5-0.x86_64 (kubernetes)
               kubernetes-cni = 0.7.5-0
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

What you expected to happen: Expected these to all install successfully as suggested by the documentation and based on this working prior to today.

How to reproduce it (as minimally and precisely as possible):

Follow these docs with a fresh centos 7 machine: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ use pinned versions of the binaries to 1.15

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

yum install -y kubelet-1.15* kubeadm-1.15* --disableexcludes=kubernetes

Anything else we need to know?: This worked yesterday before the CVE patches today. This also works fine without the kuberenetes-cni dependency on ubuntu / debian (your apt packages) and with different versions of these binaries (eg. 1.16, 1.17. etc.)

Edit: Per comments below, this does look to affect certain patch levels of 1.17 and 1.18 as well, just not the latest patch of those versions. Might also be affecting certain apt installations, but not the 1.15 latest patch that we are using.

Here is what the repo claims are dependencies of these different versions:

# repoquery --requires --resolve kubelet-1.18
ethtool-2:4.8-10.el7.x86_64
iptables-0:1.4.21-34.el7.x86_64
util-linux-0:2.23.2-63.el7.x86_64
iptables-0:1.4.21-34.el7.i686
util-linux-0:2.23.2-63.el7.i686
conntrack-tools-0:1.4.4-7.el7.x86_64
iproute-0:4.11.0-25.el7_7.2.x86_64
ebtables-0:2.0.10-16.el7.x86_64
glibc-0:2.17-307.el7.1.i686
glibc-0:2.17-307.el7.1.x86_64
socat-0:1.7.3.2-2.el7.x86_64

# repoquery --requires --resolve kubelet-1.15*
ethtool-2:4.8-10.el7.x86_64
iptables-0:1.4.21-34.el7.x86_64
util-linux-0:2.23.2-63.el7.x86_64
iptables-0:1.4.21-34.el7.i686
util-linux-0:2.23.2-63.el7.i686
kubernetes-cni-0:0.7.5-0.x86_64
conntrack-tools-0:1.4.4-7.el7.x86_64
iproute-0:4.11.0-25.el7_7.2.x86_64
ebtables-0:2.0.10-16.el7.x86_64
glibc-0:2.17-307.el7.1.i686
glibc-0:2.17-307.el7.1.x86_64
socat-0:1.7.3.2-2.el7.x86_64

Notice kuberenets-cni showing with kubelet-1.15

Additionally, if you try to install that kuberenetes-cni version, it needs kubelet and cannot install with it ❗

# repoquery --requires --resolve kubernetes-cni-0:0.7.5-0.x86_64
kubelet-0:1.6.12-1.x86_64

# yum install kubelet-1.15* kubernetes-cni
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: repos.dfw.quadranet.com
 * extras: linux-mirrors.fnal.gov
 * updates: mirrors.raystedman.org
Package kubernetes-cni is obsoleted by kubelet, trying to install kubelet-1.18.4-0.x86_64 instead
Resolving Dependencies
--> Running transaction check
---> Package kubelet.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubelet-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
--> Processing Dependency: iptables >= 1.4.21 for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: socat for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: iproute for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: ethtool for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: ebtables for package: kubelet-1.15.12-0.x86_64
--> Processing Dependency: conntrack for package: kubelet-1.15.12-0.x86_64
---> Package kubelet.x86_64 0:1.18.4-0 will be installed
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-7.el7 will be installed
--> Processing Dependency: libnetfilter_conntrack >= 1.0.6 for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libmnl.so.0(LIBMNL_1.0)(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnfnetlink.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libnetfilter_conntrack.so.3()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
--> Processing Dependency: libmnl.so.0()(64bit) for package: conntrack-tools-1.4.4-7.el7.x86_64
---> Package ebtables.x86_64 0:2.0.10-16.el7 will be installed
---> Package ethtool.x86_64 2:4.8-10.el7 will be installed
---> Package iproute.x86_64 0:4.11.0-25.el7_7.2 will be installed
---> Package iptables.x86_64 0:1.4.21-34.el7 will be installed
---> Package kubelet.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubelet-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed
--> Processing Dependency: libwrap.so.0()(64bit) for package: socat-1.7.3.2-2.el7.x86_64
--> Running transaction check
---> Package kubelet.x86_64 0:1.15.12-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubelet-1.15.12-0.x86_64
Package kubernetes-cni is obsoleted by kubelet, but obsoleting package does not provide for requirements
---> Package libmnl.x86_64 0:1.0.3-7.el7 will be installed
---> Package libnetfilter_conntrack.x86_64 0:1.0.6-1.el7_3 will be installed
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
---> Package libnfnetlink.x86_64 0:1.0.1-4.el7 will be installed
---> Package tcp_wrappers-libs.x86_64 0:7.6-77.el7 will be installed
--> Finished Dependency Resolution
Error: Package: kubelet-1.15.12-0.x86_64 (kubernetes)
           Requires: kubernetes-cni >= 0.7.5
           Available: kubernetes-cni-0.3.0.1-0.07a8a2.x86_64 (kubernetes)
               kubernetes-cni = 0.3.0.1-0.07a8a2
           Available: kubernetes-cni-0.5.1-0.x86_64 (kubernetes)
               kubernetes-cni = 0.5.1-0
           Available: kubernetes-cni-0.5.1-1.x86_64 (kubernetes)
               kubernetes-cni = 0.5.1-1
           Available: kubernetes-cni-0.6.0-0.x86_64 (kubernetes)
               kubernetes-cni = 0.6.0-0
           Available: kubernetes-cni-0.7.5-0.x86_64 (kubernetes)
               kubernetes-cni = 0.7.5-0
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

Installing kuberenets-cni on it’s own tries to install kubelet / kubeadm 1.18 which we don’t want… So there is currently NO way I can find to install kubelet-1.15 😢

Environment:

  • Kubernetes version (use kubectl version): 1.15*
  • Cloud provider or hardware configuration: any centos (AWS Ec2, bare metal, docker)
  • OS (e.g: cat /etc/os-release):
# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Kernel (e.g. uname -a):
# uname -a
Linux b80a0f2be28d 4.15.0-99-generic #100-Ubuntu SMP Wed Apr 22 20:32:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: yum
  • Network plugin and version (if this is a network-related bug):
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 21
  • Comments: 64 (18 by maintainers)

Commits related to this issue

Most upvoted comments

Seeing this in all 1.16.x, 1.17.x and 1.18.x versions that are not the latest patches.

As a workaround until the repo is fixed, you can use the --setopt=obsoletes=0 option when running yum install.

+32
dkoshkin on Jun 18, 2020

As a general note, Kubernetes 1.15 is out of support.

Yes, but I would appreciate if you don’t break 1.15 for those of us who are still on that version for the time being.

+10
blurpy on Jun 18, 2020

+1 blocking installation for lot of servers

+3
hklohar on Jun 18, 2020

Got the same errors on 1.17.4, 1.17.6 and 1.18.3

+3
ju187 on Jun 18, 2020

All – We’ve published new packages that unbundle the CNI plugins from the kubelet, as well as updated the kubernetes-cni package to 0.8.6. Can a few of you test your installation flows again and report back?

Thanks for your patience as we work through this!

Worklog from Slack: https://kubernetes.slack.com/archives/CJH2GBF7Y/p1592947121041400

+2
justaugustus on Jun 24, 2020

As a general note, Kubernetes 1.15 is out of support.

Yes, but I would appreciate if you don’t break 1.15 for those of us who are still on that version for the time being.

@blurpy – Apologies for the brief response. What I was trying to imply is that those who are able to move to a more recent version of Kubernetes are likely to have better luck with this issue. I was also using that message to inquire if only older versions were affected or newer versions as well.

It is never our intention to impair a user’s ability to consume Kubernetes. In fact, we opted to bundle the CNI plugins into the kubelet package in an attempt to minimize/prevent breakages for users on older versions.

I hope you can appreciate this was a misconfiguration and mistakes can happen. We’re all human after all.

I’ve opened a potential fix in https://github.com/kubernetes/release/pull/1375. The team will discuss ~tomorrow~ later today and make a determination on path forward.

@saschagrunert @justaugustus - Just checking in to see if you have any updates on this bug or an ETA on the formal fix? Thanks in advance.

Unfortunately not a concrete ETA but we’ll tackle that issue very soon. Probably this week, sorry for the circumstances.

As @saschagrunert mentioned, we’re hoping to have a fix to you all by the end of the week.

+2
justaugustus on Jun 23, 2020

@saschagrunert @justaugustus - Just checking in to see if you have any updates on this bug or an ETA on the formal fix? Thanks in advance.

Unfortunately not a concrete ETA but we’ll tackle that issue very soon. Probably this week, sorry for the circumstances.

+2
saschagrunert on Jun 22, 2020

@saschagrunert @justaugustus - Just checking in to see if you have any updates on this bug or an ETA on the formal fix? Thanks in advance.

+2
tstraley on Jun 22, 2020

@justaugustus ran into the same issue trying to build CAPZ images for v1.16.11, v1.17.7 and v1.18.4

+2
CecileRobertMichon on Jun 18, 2020

+1 👀

+2
matthew-mcdermott on Jun 18, 2020

Thanks for the fix. Just successfully installed 1.17.3

+1
phillewis on Jun 24, 2020

Thank you for the update. I tried installing the latest version. yum install -y kubelet kubeadm kubectl It worked absolutely fine.

Installed:
  kubeadm.x86_64 0:1.18.4-1                                                      kubectl.x86_64 0:1.18.4-1                                                      kubelet.x86_64 0:1.18.4-1

Dependency Installed:
  conntrack-tools.x86_64 0:1.4.4-7.el7         cri-tools.x86_64 0:1.13.0-0         libnetfilter_cthelper.x86_64 0:1.0.0-11.el7         libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7         libnetfilter_queue.x86_64 0:1.0.2-2.el7_2
+1
sampadsaha5 on Jun 24, 2020

UPDATE: We’ve chosen https://github.com/kubernetes/release/pull/1375 as the path forward and that PR has merged.

While we (SIG Release) maintain the debian package definitions and rpm specs, the actual packages have to be built by a specific team at Google to be published to the apt/yum repos (which are Google-owned).

I’m coordinating this now and will continue to send updates throughout the week as we move closer to resolution.

+1
justaugustus on Jun 23, 2020

we are also facing same issue with 1.17.5 version of kubelet.

+1
sujithvs-dev on Jun 18, 2020
Read more comments on GitHub
← kubernetes: kubeadm 1.8.0 init fails with "/var/lib/kubelet is not empty"
kubernetes: kubeadm blocks waiting for 'control plane' →