cilium: Unable to create map ... operation not permitted on some Ubuntu kernels with Cilium v1.6

Environment

  • Ubuntu 18.04 (with later kernel 5.0)
  • Kubernetes 1.15.0 / 1.16.0
  • Cilium 1.6.3
  • Following guide for external etcd

Symptoms

From slack thread:

Hello I am trying to use cilium 1.6.3 using external etcd but the cilium pod kept crashing due to permission issue of the bpf filesystem. I con confirm the bpf is mounted properly and the cilium yaml gives privileged context for it. sytem is runing on kernal 5.0. any other issue may cause this?

level=error msg="Error while opening/creating BPF maps" error="Unable to create map /sys/fs/bpf/tc/globals/cilium_lxc: operation not permitted" subsys=daemon
level=fatal msg="Error while creating daemon" error="Unable to create map /sys/fs/bpf/tc/globals/cilium_lxc: operation not permitted" subsys=daemon
$ mount |grep bpf
bpffs on /sys/fs/bpf type bpf (rw,relatime)

sorry it is a kernel issue. once updated to 5.2.4 it works

[Kernel package] 5.0.0-31-generic has issues with bpf fs; 5.2.4-050204-generic works fine

Common mitigations

  • Some older kernels exhibit this problem. Update to the latest version for your distribution.
  • Minikube on CentOS can hit issues with SELinux. Options:
    • Run minikube using a VM driver as a backend
    • Disable SELinux

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 35 (21 by maintainers)

Most upvoted comments

Ok, meaning you’ve seen this once on 5.4.46-23.77.amzn2 but other instances with the same kernel seemed to be fine. Also reboot seemed to have ‘fixed’ the issue, correct?

No, not really, I’ve seen it with 5.4.38-17.76.amzn2.x86_64. Upgrading to 5.4.46-23.77.amzn2(and subsequently rebooting) is what seems to have fixed the issue.

I have no enough knowledge to help on the first approach. So I’ll try to do something with the second one.

met the same issue with centos7(5.7.11-1.el7.elrepo.x86_64), minikube(v1.12.2). but after disable selinux(setenforce 0), pods get up.

maybe the link can help, https://github.com/iovisor/bcc/issues/2525