libs: BPF issue on amazon linux 2 since we upgraded from 0.29.1 to 0.30 (not working on all kernel 4.1X and 5.X and using clang 7 or clang 11)
Describe the bug We are encountering issue with the BPF module since we upgrade from falco 0.29.1 to 0.30. We are building the bpf probe using our own docker image (as an init container), we have been using the default clang llvm version for long (11) and we had to switch to clang7 since 0.29 if my memory is correct. But now, it does not seems to work with both clang version We are getting some stacktrace like this
math between map_value pointer and register with unbounded min value is not allowed
2021-11-15T15:22:14+0000: Runtime error: bpf_load_program() err=22 event=filler/sys_read_x message=0: (bf) r6 = r1
How to reproduce it Using EKS v1.18 with amazon linux 2 and clang 7 or clang 11 (latest from amazon repo)
Environment OS: amazon linux 2 (kernel : 4.14.219-161.340.amzn2.x86_64 but we also use 5.X kernel and the issue is the same) Using EKS (AWS kubernetes) 1.18 Clang + LLVM : 7 and 11 (from amazon package repo)
- Falco version:
- 0.30
- System info:
- Cloud provider or hardware configuration:
- OS:
- Kernel:
- Installation method:
Additional context We also tried to use the latest version of this repository, especially following this issue
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 24 (13 by maintainers)
Hi @JoupainMD good news: i was able to reproduce your issue! I am currently testing a fix 🤞
WORKING on 4.14.219-161.340.amzn2.x86_64 👍 💯 Thanks @FedeDP and @leogr I will now try on our 5.X kernels as well 🙏
Nice! So the good thing is we found the guilty function. Bad thing is that now i will have to provide “pseudo random” patches to try to fix it 😃 I will come back with some patches to test! Thank you!
Yep you are right, already here in 0.29.1, we didn’t notice (only on 5.X kernels it looks like).
Working on 5.4.149-73.259.amzn2.x86_64 as well. Perfect, thank you again guys 🙏 Ok for me to close this issue whenever you want.
Ok that’s clear I’ll try that asap I’ll keep you posted 🙏
This is an error coming from libsinsp, ie: it has nothing to do with kernel eBPF verifier (that is now satisfied!).
The issue is you are using falco v0.30 with new libs; you can try to backport my fix to libs shipped with falco v0.30; it should do the trick!
Hey @JoupainMD
Could you provide us the Dockerfile (or the docker image) you have used as init container? I think this is the only way for us to exactly reproduce your issue so we can then debug it. Currently, I haven’t been able to reproduce it.
Thanks in advance! 🙏
Once the kernel verifier issue is fixed, we will try to understand the error you are getting about the invalid filler. It should not be something to worry about though!
Edit: btw thanks for your time!
I did revert from master the file filler_helpers.h. But for the libs we are using commit version 1ed3e2a15dad1347459f1d55838bbbb8ae352266 as mentioned in the cmake command ahead. I will check again to be sure I did not miss anything or run the wrong container version.
About potential old bpf probe version asked by falco this is possible, we are using falco 0.30 release (not master) but libs currently we fetch+build from commit 1ed3e2a15dad1347459f1d55838bbbb8ae352266, to bypass probe version check, in our dockerfile we do override the PROBE_VERSION variable from /falco-repo/cmake/modules/falcosecurity-libs.cmake in order to match the version that falco 0.30 is wanting (in this case this is commit 3aa7a83bf7b9e6229a3824e3fd1f4452d1e95cb4) Is it really crappy ? If so I can try to build falco from master also but we really plan to use falco from 0.30 and not build it from source on production.