KubeArmor: Permission denied when KubeArmor attaches kprobes

Bug Report

General Information

Environment description (GKE, VM-Kubeadm, vagrant-dev-env, minikube, microk8s, …) vagrant-dev (NETNEXT=0)
Kernel version (run uname -a) 4.15.0-159-generic
Orchestration system version in use (e.g. kubectl version, …) v1.21.5

To Reproduce

create a clean vagrant setup

$ cd ~/KubeArmor/KubeArmor
$ make vagrant-up

get into the vagrant VM

$ make vagrant-ssh

apply KubeArmor into Kubernetes

$ cd ~/KubeArmor/deployments/generic
$ kubectl apply -f .

check the KubeArmor logs

$ kubectl -n kube-system logs kubearmor-XXXX -f

There will be no issue at this time.

delete KubeArmor

$ cd ~/KubeArmor/deployments/generic
$ kubectl delete -f .

apply KubeArmor again

$ cd ~/KubeArmor/deployments/generic
$ kubectl apply -f .

check the KubeArmor logs

$ kubectl -n kube-system logs kubearmor-YYYY -f

You will see “create_probe_event: open(/sys/kernel/debug/tracing/kprobe_events): Permission denied”.

2021-10-06 06:28:46.090869      INFO    Initializing an eBPF program
2021-10-06 06:28:54.132097      INFO    Initialized the eBPF program
2021-10-06 06:28:54.373941      ERROR   Failed to initialize the system monitor
github.com/kubearmor/KubeArmor/KubeArmor/log.Err
        /usr/src/KubeArmor/KubeArmor/log/logger.go:97
github.com/kubearmor/KubeArmor/KubeArmor/feeder.(*Feeder).Err
        /usr/src/KubeArmor/KubeArmor/feeder/feeder.go:465
github.com/kubearmor/KubeArmor/KubeArmor/core.KubeArmor
        /usr/src/KubeArmor/KubeArmor/core/kubeArmor.go:374
main.main
        /usr/src/KubeArmor/KubeArmor/main.go:49
runtime.main
        /usr/local/go/src/runtime/proc.go:204
create_probe_event: open(/sys/kernel/debug/tracing/kprobe_events): Permission denied
2021-10-06 06:28:54.401262      INFO    Stopped the system monitor

Then, KubeArmor will be terminated.

The key point is that KubeArmor can attach kprobes one time, but it cannot attach them after that. Note that when we directly run KubeArmor, this problem cannot be observed, which means that this problem only happens when we deploy KubeArmor as a container.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 17 (12 by maintainers)

Most upvoted comments

by the way, the deployment issue may be another issue. let’s clearly separate these issues and focus on this permission denied issue.

nam-jaehyun on Oct 7, 2021

@oneiro-naut Thanks! Good observation.

nam-jaehyun on Oct 11, 2021

@nam-jaehyun @nyrahul @geyslan There is a strange unconfined apparmor profile being created each time we follow the steps. It has the contents of Kubearmor default container profile:

vagrant@kubearmor-dev-next:~$ cat /etc/apparmor.d/unconfined 
## == Managed by KubeArmor == ##

#include <tunables/global>

profile unconfined flags=(attach_disconnected,mediate_deleted) {
  ## == PRE START == ##
  #include <abstractions/base>
  umount,
  file,
  network,
  capability,
  ## == PRE END == ##

  ## == POLICY START == ##
  ## == POLICY END == ##

  ## == POST START == ##
  /lib/x86_64-linux-gnu/{*,**} rm,

  deny @{PROC}/{*,**^[0-9*],sys/kernel/shm*} wkx,
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/mem rwklx,
  deny @{PROC}/kmem rwklx,
  deny @{PROC}/kcore rwklx,

  deny mount,

  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/efi/efivars/** rwklx,
  deny /sys/kernel/security/** rwklx,
  ## == POST END == ##
}

Corresponding kubectl logs for this event is:

2021-10-11 07:01:08.454985	INFO	Detected a Pod (added/kube-system/kubearmor-relay-645667c695-h4sn5)
2021-10-11 07:01:08.498292	INFO	Registered an AppArmor profile (unconfined)

This profile is causing trouble when we restart the kubearmor pods. Because runc is attaching this named unconfined profile to the necessary kubearmor processes(which includes ./entrypoint.sh). This is also causing runc to be unable to terminate pods because the named profile unconfined doesn’t have any signal related rules. Everything that is supposed to be unconfined is now having this named unconfined profile attached to it.

vagrant@kubearmor-dev-next:~$ sudo aa-unconfined --paranoid | grep unconfined
133335 /pause confined by 'unconfined (enforce)'
133784 /bin/bash (/bin/bash /KubeArmor/entrypoint.sh -gRPC=32767 -logPath=/tmp/kubearmor.log -enableKubeArmorHostPolicy) confined by 'unconfined (enforce)'
133808 /KubeArmor/kubearmor confined by 'unconfined (enforce)'

I think we avoid creating a profile with name unconfined in our code. https://github.com/kubearmor/KubeArmor/blob/db01d98dfd2669a628648f8b66849706e39919c5/KubeArmor/enforcer/appArmorEnforcer.go#L458 But somehow it’s not working that way.

oneiro-naut on Oct 11, 2021

@geyslan As far as I observed, I don’t think so. If you remove “-enableKubeArmorHostPolicy” in kubearmor.yaml, kubearmor.host is not installed. For better clarification, you can test the image after removing the above option.

nam-jaehyun on Oct 7, 2021

As I mentioned that, it happened when I used the KubeArmor deployment (not direct execution).

nam-jaehyun on Oct 7, 2021

I’ll try to reproduce this.

geyslan on Oct 6, 2021