kubernetes: kube-proxy in IPVS mode does not work on containerd with AppArmor runtime default profile
What happened: Installed a new node (joining existing cluster) with Kubernetes v1.13.4 using kubeadm and containerd v1.2.4 as the container runtime. kube-proxy was configured in IPVS mode and crashed immediately with the following error:
F0308 22:32:25.017346 1 server.go:396] unable to create proxier: can't set sysctl net/ipv4/vs/conntrack: open /proc/sys/net/ipv4/vs/conntrack: permission denied
It looks like containerd is blocking writes to procfs and sysfs.
What you expected to happen: kube-proxy starts like it does with docker-ce (v18.6.3).
How to reproduce it (as minimally and precisely as possible):
Install Linux VM with containerd instead of docker-ce as container runtime, then kubeadm join
an existing cluster with kube-proxy set up as a DaemonSet.
Anything else we need to know?: A workaround is to set up kernel modules params and sysctl settings manually:
Workaround
$ cat /etc/modprobe.d/kubernetes-cri.conf
options nf_conntrack hashsize=32768
$ cat /etc/modules-load.d/kubernetes-cri.conf
br_netfilter
nf_conntrack
nf_conntrack_ipv4
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
$ cat /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.route_localnet=1
net.ipv4.ip_forward=1
net.ipv4.vs.conn_reuse_mode=0
net.ipv4.vs.conntrack=1
net.ipv4.vs.expire_nodest_conn=1
net.ipv4.vs.expire_quiescent_template=1
net.netfilter.nf_conntrack_max=131072
net.netfilter.nf_conntrack_tcp_timeout_close_wait=3600
net.netfilter.nf_conntrack_tcp_timeout_established=86400
Environment:
- Kubernetes version (use
kubectl version
): v1.13.4 - Cloud provider or hardware configuration: VMware vSphere
- OS (e.g:
cat /etc/os-release
): Debian 9 (stretch) - Kernel (e.g.
uname -a
): Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64 GNU/Linux - Install tools: kubeadm
- Others: Containerd v1.2.4
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 20 (19 by maintainers)
@cmluciano turns out we (containerd) are doing the masks correctly… They were being set to nil if privileged a dozen or so lines down from where I looked. My tests confirm we are not applying these requested masks if the container is privileged. So this needs more debug.