falco: Falco crash at startup when using ebpf driver

Describe the bug

I’m migrating from kernel-module driver to ebpf driver (I compile it and mount the driver into falco pods) but falco crashes when starting up on ebpf mode.

How to reproduce it

Compile the ebpf driver using falco-builder image:

cmake -DBUILD_BPF=ON -DUSE_BUNDLED_DEPS=ON ..

make bpf

Then start a pod with falco 0.29.1 mounting the generated probe.o (even though doc in https://falco.org/docs/getting-started/source/#enable-bpf-support shows it should be falco.o) file into falco pod as /root/.falco/falco_centos_4.18.0-305.12.1.el8_4.x86_64_1.o

Expected behaviour

No crash

Screenshots

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.29.1, driver version=17f5df52a7d9ed6bb12d3b1768460def8439936d
* Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
* Skipping download, eBPF probe is already present in /root/.falco/falco_centos_4.18.0-305.12.1.el8_4.x86_64_1.o
* Skipping compilation, eBPF probe is already present in /root/.falco/falco_centos_4.18.0-305.12.1.el8_4.x86_64_1.o
* eBPF probe located in /root/.falco/falco_centos_4.18.0-305.12.1.el8_4.x86_64_1.o
* Success: eBPF probe symlinked to /root/.falco/falco-bpf.o
Thu Oct 21 07:54:16 2021: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d)
Thu Oct 21 07:54:16 2021: Falco initialized with configuration file /etc/falco/falco.yaml
Thu Oct 21 07:54:16 2021: Loading rules from file /etc/falco/falco_rules.yaml:
Thu Oct 21 07:54:17 2021: Loading rules from file /etc/falco/falco_rules.local.yaml:
Thu Oct 21 07:54:17 2021: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
0: (85) call bpf_get_smp_processor_id#8
1: (63) *(u32 *)(r10 -4) = r0
2: (bf) r2 = r10
3: (07) r2 += -4
4: (18) r1 = 0xffff9bb7ac3fdc00
6: (85) call bpf_map_lookup_elem#1
7: (15) if r0 == 0x0 goto pc+67
 R0_w=map_value(id=0,off=0,ks=4,vs=77,imm=0) R10=fp0 fp-8=mmmm????
8: (71) r1 = *(u8 *)(r0 +37)
 R0_w=map_value(id=0,off=0,ks=4,vs=77,imm=0) R10=fp0 fp-8=mmmm????
9: (67) r1 <<= 8
10: (71) r2 = *(u8 *)(r0 +36)
 R0_w=map_value(id=0,off=0,ks=4,vs=77,imm=0) R1_w=inv(id=0,umax_value=65280,var_off=(0x0; 0xff00)) R10=fp0 fp-8=mmmm????
11: (4f) r1 |= r2
12: (71) r3 = *(u8 *)(r0 +38)
 R0_w=map_value(id=0,off=0,ks=4,vs=77,imm=0) R1_w=inv(id=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0 fp-8=mmmm????
13: (71) r2 = *(u8 *)(r0 +39)
 R0_w=map_value(id=0,off=0,ks=4,vs=77,imm=0) R1_w=inv(id=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R3_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0 fp-8=mmmm????
14: (67) r2 <<= 8
15: (4f) r2 |= r3
16: (67) r2 <<= 16
17: (4f) r2 |= r1
18: (18) r1 = 0xfffffffd
20: (2d) if r1 > r2 goto pc+50
 R0=map_value(id=0,off=0,ks=4,vs=77,imm=0) R1=inv4294967293 R2=inv(id=0,smin_value=-9223372032559808516,umin_value=4294967293,var_off=(0xfffffffc; 0xffffffff00000003),s32_min_value=-3,s32_max_value=-1) R3=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0 fp-8=mmmm????
21: (07) r2 += 3
22: (67) r2 <<= 32
23: (77) r2 >>= 32
24: (67) r2 <<= 3
25: (bf) r1 = r0
26: (1f) r1 -= r2
last_idx 26 first_idx 20
regs=4 stack=0 before 25: (bf) r1 = r0
regs=4 stack=0 before 24: (67) r2 <<= 3
regs=4 stack=0 before 23: (77) r2 >>= 32
regs=4 stack=0 before 22: (67) r2 <<= 32
regs=4 stack=0 before 21: (07) r2 += 3
regs=4 stack=0 before 20: (2d) if r1 > r2 goto pc+50
 R0_rw=map_value(id=0,off=0,ks=4,vs=77,imm=0) R1_rw=inv4294967293 R2_rw=invP(id=0) R3_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0 fp-8=mmmm????
parent didn't have regs=4 stack=0 marks
last_idx 18 first_idx 0
regs=4 stack=0 before 18: (18) r1 = 0xfffffffd
regs=4 stack=0 before 17: (4f) r2 |= r1
regs=6 stack=0 before 16: (67) r2 <<= 16
regs=6 stack=0 before 15: (4f) r2 |= r3
regs=e stack=0 before 14: (67) r2 <<= 8
regs=e stack=0 before 13: (71) r2 = *(u8 *)(r0 +39)
regs=a stack=0 before 12: (71) r3 = *(u8 *)(r0 +38)
regs=2 stack=0 before 11: (4f) r1 |= r2
regs=6 stack=0 before 10: (71) r2 = *(u8 *)(r0 +36)
regs=2 stack=0 before 9: (67) r1 <<= 8
regs=2 stack=0 before 8: (71) r1 = *(u8 *)(r0 +37)
27: (71) r2 = *(u8 *)(r1 +65)
 R0=map_value(id=0,off=0,ks=4,vs=77,imm=0) R1_w=map_value(id=0,off=0,ks=4,vs=77,smin_value=-16,smax_value=0,umax_value=18446744073709551608,var_off=(0x0; 0xfffffffffffffff8),s32_max_value=2147483640,u32_max_value=-8) R2_w=invP(id=0,umax_value=16,var_off=(0x0; 0x18),s32_max_value=24,u32_max_value=24) R3=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0 fp-8=mmmm????
R1 unbounded memory access, make sure to bounds check any such access
processed 26 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1
Thu Oct 21 07:53:25 2021: Runtime error: bpf_load_program() err=13 event=filler/terminate_filler message=0: (85) call bpf_get_smp_processor_id#8
1: (63) *(u32 *)(r10 -4) = r0
2: (bf) r2 = r10
3: (07) r2 += -4
4: (18) r1 = 0xffff9bb9f23f2000
6: (85) call bpf_map_lookup_elem#1
7: (15) if r0 == 0x0. Exiting.

Environment

Kubernetes 1.20.10

Falco version: 0.29.1
System info: { “machine”: “x86_64”, “nodename”: “74e2a2435bbc”, “release”: “4.18.0-305.12.1.el8_4.x86_64”, “sysname”: “Linux”, “version”: “#1 SMP Wed Aug 11 01:59:55 UTC 2021” }
Cloud provider or hardware configuration:
OS: CentOS Linux release 8.4.210
Kernel: Linux 4.18.0-305.12.1.el8_4.x86_64 #1 SMP Wed Aug 11 01:59:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Installation method: Kubernetes

Additional context

Arguments passed to falco on start:

        - /usr/bin/falco
        - '--cri'
        - /run/containerd/containerd.sock
        - '-K'
        - /var/run/secrets/kubernetes.io/serviceaccount/token
        - '-k'
        - 'https://kubernetes.default:443'
        - '-pk'
        - '--disable-cri-async'

This issue could be related to #1690 but the failure is not the same.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 17 (10 by maintainers)

Most upvoted comments

The libs PR is now merged thus i think we can consider this solved 😉 Again, thank you very much for your help, much appreciated!

FedeDP on Oct 29, 2021

Good news! I’m an idiot XD

I’ve been updating the wrong ppm_flag_helpers.h file, when I execute the cmake command, 2 files named like that are created:

./build/driver/src/ppm_flag_helpers.h
./build/falcosecurity-libs-repo/falcosecurity-libs-prefix/src/falcosecurity-libs/driver/ppm_flag_helpers.h

I’ve updated the first one, not the second one (I didn’t see it the first time and all the tests have been performed using my bash shell history)

Now I’ve applied your PR patch correctly and it works absolutely fine 😃

Thanks a lot for your work!!

jasiam on Oct 28, 2021

Hi!

I’ve just compiled probe.o with your new patch in a falco-builder container and I’ve got a new bpf error when falco starts up in my environment. You can check the error trace in this gist

Regarding init error messages about not able to compile bpf probe, be aware that I’m mounting the compiled probe.o as /root/.falco/falco-bpf.o in my falco container.

Thank you for your bpf knowledge @FedeDP!

jasiam on Oct 28, 2021

I’ve just tested using clang-11 (the easiest version to install in my environment) and it’s working fine apparently 😃

So if I’ve understood correctly, official falcosecurity/falco-builder:latest image is not valid if you want to build bpf probe. Am I right? Should I open a new issue for this?

Thanks!

jasiam on Oct 25, 2021

Hi @FedeDP! Thanks for the note about the ebpf verifier. I got the same failure with 0.30.0, but checking your PR I’ve noticed that falco-builder:latest docker image (which I use to compile the ebpf driver) includes clang version 5.0.1 (tags/RELEASE_501/final) Could it be the reason of the compiled code being rejected by the ebpf verifier? Should it be bumped to clang 7?

jasiam on Oct 22, 2021