falco: Cannot run falco "Least Privileged", pmu_fd: operation not permitted

Describe the bug

Falco container fails to start with error: Error: pmu_fd: Operation not permitted

See Logs section below for more info.

How to reproduce it

  1. Install falco on AKS (1.25.6) using helm chart version 3.1.3 and following values:
  podAnnotations: 
    container.apparmor.security.beta.kubernetes.io/falco: unconfined
  tty: false
  controller:
    kind: daemonset
  driver:
    enabled: true
    kind: ebpf
    ebpf:
      hostNetwork: true
      leastPrivileged: true
  collectors:
    enabled: true
  1. Notice the error in the logs and the crashing pods

Expected behaviour

Falco runs without error

Logs

Logs of the starting pod, Falco:

Tue Apr 11 21:41:52 2023: Falco version: 0.34.1 (x86_64)
Tue Apr 11 21:41:52 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Tue Apr 11 21:41:52 2023: Loading rules from file /etc/falco/falco_rules.yaml
Tue Apr 11 21:41:52 2023: Loading rules from file /etc/falco/rules.d/custom.local.yaml
Tue Apr 11 21:41:52 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Tue Apr 11 21:41:52 2023: gRPC server threadiness equals to 4
Tue Apr 11 21:41:52 2023: Starting health webserver with threadiness 4, listening on port 8765
Tue Apr 11 21:41:52 2023: Enabled event sources: syscall
Tue Apr 11 21:41:52 2023: Opening capture with BPF probe. BPF probe path: /root/.falco/falco-bpf.o
Tue Apr 11 21:41:52 2023: Starting gRPC server at unix:///run/falco/falco.sock
Tue Apr 11 21:41:52 2023: An error occurred in an event source, forcing termination...
Tue Apr 11 21:41:52 2023: Shutting down gRPC server. Waiting until external connections are closed by clients
Tue Apr 11 21:41:52 2023: Waiting for the gRPC threads to complete
Tue Apr 11 21:41:52 2023: Draining all the remaining gRPC events
Tue Apr 11 21:41:52 2023: Shutting down gRPC server complete
Error: pmu_fd: Operation not permitted
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:

Logs of the driver loader

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.34.1, driver version=4.0.0+driver, arch=x86_64, kernel release=5.15.0-1034-azure, kernel version=41
* Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
mount: /sys/kernel/debug: cannot mount nodev read-only.
* Filename 'falco_ubuntu-azure_5.15.0-1034-azure_41.o' is composed of:
 - driver name: falco
 - target identifier: ubuntu-azure
 - kernel release: 5.15.0-1034-azure
 - kernel version: 41
* Trying to download a prebuilt eBPF probe from https://download.falco.org/driver/4.0.0%2Bdriver/x86_64/falco_ubuntu-azure_5.15.0-1034-azure_41.o
* Skipping compilation, eBPF probe is already present in /root/.falco/4.0.0+driver/x86_64/falco_ubuntu-azure_5.15.0-1034-azure_41.o
* eBPF probe located in /root/.falco/4.0.0+driver/x86_64/falco_ubuntu-azure_5.15.0-1034-azure_41.o
* Success: eBPF probe symlinked to /root/.falco/falco-bpf.o

Environment

Additional context

I tested with both no AppArmor configuration and with unconfined given the notice mentioned in the doc : https://falco.org/docs/getting-started/running/#docker-least-privileged. However, both configurations provide the same error.

Also, falco works fine if using leastPrivileged: false, here is an example of normal logs:

Tue Apr 11 22:04:35 2023: Falco version: 0.34.1 (x86_64)
Tue Apr 11 22:04:35 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Tue Apr 11 22:04:35 2023: Loading rules from file /etc/falco/falco_rules.yaml
Tue Apr 11 22:04:35 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Tue Apr 11 22:04:35 2023: Starting health webserver with threadiness 4, listening on port 8765
Tue Apr 11 22:04:35 2023: Enabled event sources: syscall
Tue Apr 11 22:04:35 2023: Opening capture with BPF probe. BPF probe path: /root/.falco/falco-bpf.o
22:05:19.262671245: Notice Unexpected connection to K8s API Server from container (command=python -u /app/sidecar.py pid=9171 k8s.ns=monitoring k8s.pod=kube-prometheus-stack-grafana-684679fcd6-gpn2b container=541747452f5f image=quay.io/kiwigrid/k8s-sidecar:1.19.2 connection=10.176.16.103:44470->192.168.0.1:443)

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 42 (28 by maintainers)

Commits related to this issue

Most upvoted comments

It just is a bit misleading right now for end-users of the helm chart since by default it will set privileged to true for modern-bpf:

yep, that was intentional because the modern probe was an experimental feature in Falco 0.34 and we were not sure about the least privileged support, but it seems to work well so we will update the helm chart for Falco 0.35 since the modern probe will be an officially supported driver 😃!

+3
Andreagit97 on Apr 13, 2023

Related to the initial issue here with CAP_PERFMON and CAP_BPF:

Recent kernels (>=5.8) introduced new capabilities to split CAP_SYS_ADMIN into several capabilities (for example CAP_PERFMON and CAP_BPF). So if you have a recent enough kernel you should be able to run the old probe and the modern probe with new capabilities. Unfortunately, this is not as simple as it sounds… There is a second variable to take into consideration: kernel.perf_event_paranoid. Reading the manual it seems that perf_event_paranoid influences only the behavior of unprivileged users, if you have the right capabilities all should work fine

-1 - not paranoid at all
0  - disallow raw tracepoint access for unprivileged users
1  - disallow CPU events for unprivileged users
2  - disallow kernel profiling for unprivileged users

But under the hood, some distros like Debian and Ubuntu introduce other perf_event_paranoid levels, see Ubuntu here: https://kernel.ubuntu.com/git/ubuntu-stable/ubuntu-stable-jammy.git/tree/kernel/events/core.c#n11991

        err = security_perf_event_open(&attr, PERF_SECURITY_OPEN);
	if (err)
		return err;

	if (perf_paranoid_any() && !capable(CAP_SYS_ADMIN))
		return -EACCES;

where perf_paranoid_any() is

static inline bool perf_paranoid_any(void)
{
	return sysctl_perf_event_paranoid > 2;
}

As you can easily notice if your kernel.perf_event_paranoid is >2 CAP_PERFMON will be not enough, you will need CAP_SYS_ADMIN! that’s the reason why the old probe could work with CAP_PERFMON + CAP_BPF only on some nodes, probably in these nodes kernel.perf_event_paranoid is <= 2

Supposing that kernel 5.8 is the first one with CAP_BPF and CAP_PERFMON available (this could be not true it really depends on you kernel patches/backports):

old probe:

  • (Kernels < 5.8) -> needs CAP_SYS_ADMIN
  • (Kernels >= 5.8) and kernel.perf_event_paranoid <= 2 -> can use CAP_BPF and CAP_PERFMON
  • (Kernels >= 5.8) and kernel.perf_event_paranoid > 2 -> it really depends on your distro but usually needs CAP_SYS_ADMIN

modern probe:

  • (Kernels < 5.8) -> needs CAP_SYS_ADMIN
  • (Kernels >= 5.8) -> can use CAP_BPF and CAP_PERFMON (the modern probe use the BPF ring-buffer so no need of kernel.perf_event_paranoid
+3
Andreagit97 on Apr 13, 2023

@Andreagit97 I just tested it and it works like a charm! Thank you for the hard and good work!!

+2
fcrespofastly on Jun 30, 2023

Sounds about right: image

+2
jemag on Apr 14, 2023

Works great on the Ubuntu AKS cluster as well. Replied to the other issue about AzureLinux @tspearconquest

Feel free to close this one

+1
jemag on Jul 19, 2023

Thank you for the update I’m working on that!

+1
Andreagit97 on Jun 20, 2023

thanks!

+1
Andreagit97 on Jun 19, 2023
Read more comments on GitHub
← falco: "BPF doesn't have JIT enabled" reported incorrectly when using docker
falco: Combination of default and customized macro not working →