libs: eBPF Probe Compilation Fails on Amazon EKS BottleRocket
Bug Description
The falco-driver-loader init container of the falco pod tries to compile an eBPF probe & fails.
Steps to Reproduce
helm install falco falco/falco --set driver.kind=ebpf
Check logs of the falco-driver-loader init container of the falco pod.
Expected Behaviour
- Either a prebuilt eBPF probe for BottleRocket should be available for download from https://download.falco.org/
- OR the eBPF probe should compile successfully
Actual Behaviour
The falco-driver-loader init container of the falco pod fails with these logs:
* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.33.0, driver version=3.0.1+driver, arch=x86_64, kernel release=5.10.135, kernel version=1
* Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
mount: /sys/kernel/debug: permission denied.
* Filename 'falco_bottlerocket_5.10.135_1.o' is composed of:
- driver name: falco
- target identifier: bottlerocket
- kernel release: 5.10.135
- kernel version: 1
* Trying to download a prebuilt eBPF probe from https://download.falco.org/driver/3.0.1%2Bdriver/x86_64/falco_bottlerocket_5.10.135_1.o
curl: (22) The requested URL returned error: 404
Unable to find a prebuilt falco eBPF probe
* Trying to compile the eBPF probe (falco_bottlerocket_5.10.135_1.o)
exec-cmd.c:210:1: fatal error: opening dependency file /host/usr/src/kernels/5.10.135/tools/objtool/.exec-cmd.o.d: Permission denied
}
^
compilation terminated.
make[5]: *** [/host/usr/src/kernels/5.10.135/tools/build/Makefile.build:97: /host/usr/src/kernels/5.10.135/tools/objtool/exec-cmd.o] Error 1
make[4]: *** [Makefile:59: /host/usr/src/kernels/5.10.135/tools/objtool/libsubcmd-in.o] Error 2
make[3]: *** [Makefile:68: /host/usr/src/kernels/5.10.135/tools/objtool/libsubcmd.a] Error 2
make[2]: *** [Makefile:68: objtool] Error 2
make[1]: *** [Makefile:1934: tools/objtool] Error 2
make: *** [Makefile:38: all] Error 2
mv: cannot stat '/usr/src/falco-3.0.1+driver/bpf/probe.o': No such file or directory
Unable to load the falco eBPF probe
Relevant info from a BottleRocket EKS worker node:
# SSM to EKS worker node
[ssm-user@control]$ enter-admin-container
[root@admin]# sudo sheltie
bash-5.1# ls -al /usr/src/kernels/5.10.135/tools/objtool/ | grep exec
-rw-r--r--. 1 root root 5284 Nov 9 05:14 .exec-cmd.o.cmd
-rw-r--r--. 1 root root 156576 Nov 9 05:14 exec-cmd.o
Environment
Falco Helm Chart Version: 2.2.0 Falco App Version: 0.33.0
Amazon EKS Kubernetes Version: 1.23 BottleRocket AMI Version: 1.10.1-5d27ae74
# SSM to EKS worker node
[ssm-user@control]$ enter-admin-container
[root@admin]# sudo sheltie
bash-5.1# ls /etc/*-release
/etc/os-release
bash-5.1# cat /etc/os-release
NAME=Bottlerocket
ID=bottlerocket
VERSION="1.10.1 (aws-k8s-1.23)"
PRETTY_NAME="Bottlerocket OS 1.10.1 (aws-k8s-1.23)"
VARIANT_ID=aws-k8s-1.23
VERSION_ID=1.10.1
BUILD_ID=5d27ae74
HOME_URL="https://github.com/bottlerocket-os/bottlerocket"
SUPPORT_URL="https://github.com/bottlerocket-os/bottlerocket/discussions"
BUG_REPORT_URL="https://github.com/bottlerocket-os/bottlerocket/issues"
bash-5.1# uname -a
Linux ip-10-0-0-183.eu-west-1.compute.internal 5.10.135 #1 SMP Tue Oct 18 00:05:25 UTC 2022 x86_64 GNU/Linux
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 21 (14 by maintainers)
Falco Helm chart version 3.1.2 installed successfully on Amazon EKS 1.24 with Bottlerocket 1.12 worker nodes & these Helm value overrides:
Thanks for the fix everyone. Unfortunately, I won’t be able to test it at least for the next few weeks. I’ll try to test & reply as soon as I can. 🙂
I started the work to add support for bottlerocket to our infra: https://github.com/falcosecurity/kernel-crawler/pull/79. Basically, we need 3 pieces glued together:
I will keep you updated about any news!
Falco 0.34 is out, and should work out of the box on Bottlerocket! 🚀 If you are able to test, we might want to close this one 😉
(You can see currently provided driver here: https://download.falco.org/driver/site/index.html?lib=4.0.0%2Bdriver&target=bottlerocket&arch=all&kind=all)
awesome news! We will absolutely check it out as soon as 0.34 drops 😃 Thanks for the great work!
OK everything is now in place! 🚀 Drivers will start being built next monday, but your falco-driver-loader script won’t be able to retrieve them because some changes were needed to it. You will need Falco 0.34 that is expected to be released around end of january 😉
We now have bottlerocket entries in kernel-crawler generated json: https://falcosecurity.github.io/kernel-crawler/?arch=x86_64&target=BottleRocket.
Yes and I can confirm with that setup, the build does work (albeit with some compilation warnings in falco-driver-loader) and the pods come up clean in the helm chart approach. Obviously we want to try and bake this straight into the machine image to avoid privileged pods as much as possible but this is a good step forward
Not sure I saw this one
On Tue, Dec 6, 2022 at 8:43 AM Federico Di Pierro @.***> wrote:
Hi! Sorry for the long delay! @maynardflies i think you built libs master and you want to run it against Falco 0.33 libs? We added
sys_epoll_createsupport in the meantime, but Falco isn’t built with the libs that support it. So, you should build driver at the3.0.1+drivertag, that is the one used by Falco 0.33.@HarishKM7 weird! Given your output, it seems like dep file
.exec-cmd.o.dis not present:But i never saw this issue before. I will play around it to see if we need any fix!