kubernetes: kube-proxy in IPVS mode does not work on containerd with AppArmor runtime default profile

What happened: Installed a new node (joining existing cluster) with Kubernetes v1.13.4 using kubeadm and containerd v1.2.4 as the container runtime. kube-proxy was configured in IPVS mode and crashed immediately with the following error:

F0308 22:32:25.017346       1 server.go:396] unable to create proxier: can't set sysctl net/ipv4/vs/conntrack: open /proc/sys/net/ipv4/vs/conntrack: permission denied

It looks like containerd is blocking writes to procfs and sysfs.

What you expected to happen: kube-proxy starts like it does with docker-ce (v18.6.3).

How to reproduce it (as minimally and precisely as possible): Install Linux VM with containerd instead of docker-ce as container runtime, then kubeadm join an existing cluster with kube-proxy set up as a DaemonSet.

Anything else we need to know?: A workaround is to set up kernel modules params and sysctl settings manually:

Workaround
$ cat /etc/modprobe.d/kubernetes-cri.conf
options nf_conntrack hashsize=32768

$ cat /etc/modules-load.d/kubernetes-cri.conf
br_netfilter
nf_conntrack
nf_conntrack_ipv4
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh

$ cat /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.route_localnet=1
net.ipv4.ip_forward=1
net.ipv4.vs.conn_reuse_mode=0
net.ipv4.vs.conntrack=1
net.ipv4.vs.expire_nodest_conn=1
net.ipv4.vs.expire_quiescent_template=1
net.netfilter.nf_conntrack_max=131072
net.netfilter.nf_conntrack_tcp_timeout_close_wait=3600
net.netfilter.nf_conntrack_tcp_timeout_established=86400

Environment:

  • Kubernetes version (use kubectl version): v1.13.4
  • Cloud provider or hardware configuration: VMware vSphere
  • OS (e.g: cat /etc/os-release): Debian 9 (stretch)
  • Kernel (e.g. uname -a): Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64 GNU/Linux
  • Install tools: kubeadm
  • Others: Containerd v1.2.4

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 20 (19 by maintainers)

Most upvoted comments

@cmluciano turns out we (containerd) are doing the masks correctly… They were being set to nil if privileged a dozen or so lines down from where I looked. My tests confirm we are not applying these requested masks if the container is privileged. So this needs more debug.