falco: rule match causes crash "Error: rule id or priority out of bounds" in stats_manager.cpp

Running Falco version: 0.37.0 (x86_64), hitting a rule causes the falco pod to crash with “Error: rule id or priority out of bounds”

Set up observation -

$ kubectl -n falco logs falco-68ddd -f

Initial output is

Defaulted container "falco" out of: falco, falcoctl-artifact-follow, falco-driver-loader (init), falcoctl-artifact-install (init)
Wed Feb  7 11:59:02 2024: Falco version: 0.37.0 (x86_64)
Wed Feb  7 11:59:02 2024: Falco initialized with configuration file: /etc/falco/falco.yaml
Wed Feb  7 11:59:02 2024: System info: Linux version 5.4.0-170-generic (buildd@lcy02-amd64-059) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #188-Ubuntu SMP Wed Jan 10 09:51:01 UTC 2024
Wed Feb  7 11:59:02 2024: Loading rules from file /etc/falco/falco_rules.yaml
Wed Feb  7 11:59:02 2024: Loading rules from file /etc/falco/rules.d/exceptions.yaml
Wed Feb  7 11:59:02 2024: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Wed Feb  7 11:59:02 2024: Starting health webserver with threadiness 6, listening on 0.0.0.0:8765
Wed Feb  7 11:59:02 2024: Loaded event sources: syscall
Wed Feb  7 11:59:02 2024: Enabled event sources: syscall
Wed Feb  7 11:59:02 2024: Opening 'syscall' source with BPF probe. BPF probe path: /root/.falco/falco-bpf.o

Then shell into a pod running on the same node,

[Thu Feb 08 09:53:18] peter@peters-mbp:~/Downloads$ kubectl -n keycloak-alpha exec -it devops-keycloak-postgres-0 -- /bin/bash
[... banners deleted for brevity ...]

Copy a binary to somewhere it should not be, execute (thereby hitting a rule):

root@devops-keycloak-postgres-0:/home/postgres# cd
root@devops-keycloak-postgres-0:~# cp /bin/cat .
root@devops-keycloak-postgres-0:~# ./cat
.bash_history  .bashrc        .config/       .profile       cat
root@devops-keycloak-postgres-0:~# ./cat .bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
[ ... rest of output elided ...]

This produces the following log output:

{"hostname":"falco-68ddd","output":"07:05:02.351487517: Warning Sensitive file opened for reading by non-trusted program (file=/etc/shadow gparent=patroni ggparent=runsv gggparent=runsvdir evt_type=openat user=postgres user_uid=101 user_loginuid=-1 process=vacuumdb proc_exepath=/usr/bin/perl parent=post_init.sh command=vacuumdb /usr/bin/vacuumdb -aZ terminal=0 container_id=c74f6d1d11ad container_image=ghcr.io/zalando/spilo-15 container_image_tag=3.0-p1 container_name=k8s_postgres_devops-keycloak-postgres-0_keycloak-alpha_f24f3be9-4fbf-430b-8171-94898964d2fd_0 k8s_ns=keycloak-alpha k8s_pod_name=devops-keycloak-postgres-0)","priority":"Warning","rule":"Read sensitive file untrusted","source":"syscall","tags":["T1555","container","filesystem","host","maturity_stable","mitre_credential_access"],"time":"2024-02-08T07:05:02.351487517Z", "output_fields": {"container.id":"c74f6d1d11ad","container.image.repository":"ghcr.io/zalando/spilo-15","container.image.tag":"3.0-p1","container.name":"k8s_postgres_devops-keycloak-postgres-0_keycloak-alpha_f24f3be9-4fbf-430b-8171-94898964d2fd_0","evt.time":1707375902351487517,"evt.type":"openat","fd.name":"/etc/shadow","k8s.ns.name":"keycloak-alpha","k8s.pod.name":"devops-keycloak-postgres-0","proc.aname[2]":"patroni","proc.aname[3]":"runsv","proc.aname[4]":"runsvdir","proc.cmdline":"vacuumdb /usr/bin/vacuumdb -aZ","proc.exepath":"/usr/bin/perl","proc.name":"vacuumdb","proc.pname":"post_init.sh","proc.tty":0,"user.loginuid":-1,"user.name":"postgres","user.uid":101}}
Events detected: 1
Rule counts by severity:
   WARNING: 1
Triggered rules by rule name:
   Read sensitive file untrusted: 1
Error: rule id or priority out of bounds

And the pod has restarted:

[Thu Feb 08 09:54:10] peter@peters-mbp:~$ kubectl -n falco get pods -o wide
NAME                                  READY   STATUS    RESTARTS      AGE   IP              NODE          NOMINATED NODE   READINESS GATES
falco-5spm7                           2/2     Running   0             20h   10.233.65.31    kubemanet41   <none>           <none>
falco-68ddd                           2/2     Running   1 (32s ago)   20h   10.233.67.244   kubemanet44   <none>           <none>
falco-b9zld                           2/2     Running   0             20h   10.233.68.147   kubemanet45   <none>           <none>
falco-falcosidekick-bbd4bdf6c-9mmr7   1/1     Running   0             24h   10.233.67.110   kubemanet44   <none>           <none>
falco-falcosidekick-bbd4bdf6c-g4xq2   1/1     Running   0             22h   10.233.66.204   kubemanet42   <none>           <none>
falco-fz6sf                           2/2     Running   2 (12m ago)   20h   10.233.69.40    kubemanet43   <none>           <none>
falco-m6726                           2/2     Running   0             20h   10.233.66.68    kubemanet42   <none>           <none>
falco-wr4ff                           2/2     Running   0             20h   10.233.64.49    kubemanet40   <none>           <none>

Expected behaviour

Falco should report the rule match, but keep running.

Screenshots

[Thu Feb 08 09:52:25] peter@peters-mbp:~$ kubectl -n falco logs falco-68ddd -f
Defaulted container "falco" out of: falco, falcoctl-artifact-follow, falco-driver-loader (init), falcoctl-artifact-install (init)
Wed Feb  7 11:59:02 2024: Falco version: 0.37.0 (x86_64)
Wed Feb  7 11:59:02 2024: Falco initialized with configuration file: /etc/falco/falco.yaml
Wed Feb  7 11:59:02 2024: System info: Linux version 5.4.0-170-generic (buildd@lcy02-amd64-059) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #188-Ubuntu SMP Wed Jan 10 09:51:01 UTC 2024
Wed Feb  7 11:59:02 2024: Loading rules from file /etc/falco/falco_rules.yaml
Wed Feb  7 11:59:02 2024: Loading rules from file /etc/falco/rules.d/exceptions.yaml
Wed Feb  7 11:59:02 2024: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Wed Feb  7 11:59:02 2024: Starting health webserver with threadiness 6, listening on 0.0.0.0:8765
Wed Feb  7 11:59:02 2024: Loaded event sources: syscall
Wed Feb  7 11:59:02 2024: Enabled event sources: syscall
Wed Feb  7 11:59:02 2024: Opening 'syscall' source with BPF probe. BPF probe path: /root/.falco/falco-bpf.o
{"hostname":"falco-68ddd","output":"07:05:02.351487517: Warning Sensitive file opened for reading by non-trusted program (file=/etc/shadow gparent=patroni ggparent=runsv gggparent=runsvdir evt_type=openat user=postgres user_uid=101 user_loginuid=-1 process=vacuumdb proc_exepath=/usr/bin/perl parent=post_init.sh command=vacuumdb /usr/bin/vacuumdb -aZ terminal=0 container_id=c74f6d1d11ad container_image=ghcr.io/zalando/spilo-15 container_image_tag=3.0-p1 container_name=k8s_postgres_devops-keycloak-postgres-0_keycloak-alpha_f24f3be9-4fbf-430b-8171-94898964d2fd_0 k8s_ns=keycloak-alpha k8s_pod_name=devops-keycloak-postgres-0)","priority":"Warning","rule":"Read sensitive file untrusted","source":"syscall","tags":["T1555","container","filesystem","host","maturity_stable","mitre_credential_access"],"time":"2024-02-08T07:05:02.351487517Z", "output_fields": {"container.id":"c74f6d1d11ad","container.image.repository":"ghcr.io/zalando/spilo-15","container.image.tag":"3.0-p1","container.name":"k8s_postgres_devops-keycloak-postgres-0_keycloak-alpha_f24f3be9-4fbf-430b-8171-94898964d2fd_0","evt.time":1707375902351487517,"evt.type":"openat","fd.name":"/etc/shadow","k8s.ns.name":"keycloak-alpha","k8s.pod.name":"devops-keycloak-postgres-0","proc.aname[2]":"patroni","proc.aname[3]":"runsv","proc.aname[4]":"runsvdir","proc.cmdline":"vacuumdb /usr/bin/vacuumdb -aZ","proc.exepath":"/usr/bin/perl","proc.name":"vacuumdb","proc.pname":"post_init.sh","proc.tty":0,"user.loginuid":-1,"user.name":"postgres","user.uid":101}}
Events detected: 1
Rule counts by severity:
   WARNING: 1
Triggered rules by rule name:
   Read sensitive file untrusted: 1
Error: rule id or priority out of bounds

[Thu Feb 08 09:54:10] peter@peters-mbp:~$ kubectl -n falco get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES falco-5spm7 2/2 Running 0 20h 10.233.65.31 kubemanet41 <none> <none> falco-68ddd 2/2 Running 1 (32s ago) 20h 10.233.67.244 kubemanet44 <none> <none> falco-b9zld 2/2 Running 0 20h 10.233.68.147 kubemanet45 <none> <none> falco-falcosidekick-bbd4bdf6c-9mmr7 1/1 Running 0 24h 10.233.67.110 kubemanet44 <none> <none> falco-falcosidekick-bbd4bdf6c-g4xq2 1/1 Running 0 22h 10.233.66.204 kubemanet42 <none> <none> falco-fz6sf 2/2 Running 2 (12m ago) 20h 10.233.69.40 kubemanet43 <none> <none> falco-m6726 2/2 Running 0 20h 10.233.66.68 kubemanet42 <none> <none> falco-wr4ff 2/2 Running 0 20h 10.233.64.49 kubemanet40 <none> <none> [Thu Feb 08 09:54:41] peter@peters-mbp:~$

Environment

[Thu Feb 08 09:54:41] peter@peters-mbp:~$ kubectl version Client Version: v1.29.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.25.15

• Falco version:

root@falco-68ddd:/# falco --version Thu Feb 8 09:11:02 2024: Falco version: 0.37.0 (x86_64) Thu Feb 8 09:11:02 2024: Falco initialized with configuration file: /etc/falco/falco.yaml Thu Feb 8 09:11:02 2024: System info: Linux version 5.4.0-170-generic (buildd@lcy02-amd64-059) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #188-Ubuntu SMP Wed Jan 10 09:51:01 UTC 2024 {“default_driver_version”:“7.0.0+driver”,“driver_api_version”:“8.0.0”,“driver_schema_version”:“2.0.0”,“engine_version”:“31”,“engine_version_semver”:“0.31.0”,“falco_version”:“0.37.0”,“libs_version”:“0.14.2”,“plugin_api_version”:“3.2.0”}

• System info:

root@falco-68ddd:/# falco --support | jq .system_info Thu Feb 8 09:11:42 2024: Falco version: 0.37.0 (x86_64) Thu Feb 8 09:11:42 2024: Falco initialized with configuration file: /etc/falco/falco.yaml Thu Feb 8 09:11:42 2024: System info: Linux version 5.4.0-170-generic (buildd@lcy02-amd64-059) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #188-Ubuntu SMP Wed Jan 10 09:51:01 UTC 2024 Thu Feb 8 09:11:42 2024: Loading rules from file /etc/falco/falco_rules.yaml Thu Feb 8 09:11:42 2024: Loading rules from file /etc/falco/rules.d/exceptions.yaml { “machine”: “x86_64”, “nodename”: “falco-68ddd”, “release”: “5.4.0-170-generic”, “sysname”: “Linux”, “version”: “#188-Ubuntu SMP Wed Jan 10 09:51:01 UTC 2024” }

• Cloud provider or hardware configuration:
• OS:

evr-pehan@kubemanet44:~$ cat /etc/os-release NAME=“Ubuntu” VERSION=“20.04.5 LTS (Focal Fossa)” ID=ubuntu ID_LIKE=debian PRETTY_NAME=“Ubuntu 20.04.5 LTS” VERSION_ID=“20.04” HOME_URL=“https://www.ubuntu.com/” SUPPORT_URL=“https://help.ubuntu.com/” BUG_REPORT_URL=“https://bugs.launchpad.net/ubuntu/” PRIVACY_POLICY_URL=“https://www.ubuntu.com/legal/terms-and-policies/privacy-policy” VERSION_CODENAME=focal UBUNTU_CODENAME=focal

• Kernel:

evr-pehan@kubemanet44:~$ uname -a Linux kubemanet44 5.4.0-170-generic #188-Ubuntu SMP Wed Jan 10 09:51:01 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

• Installation method:

Kubernetes

Additional context

The messages all apparently come from stats_manager.cpp. My C++ is not strong enogh to supply a fix, but I suspect initialization to 0 of key variables with subsequent test for non-zeroness without any assignment other than initialization may be our culprit here.