falco: Output sometimes missing k8s pod/container information

Describe the bug I’ve installed falco using Helm, to an EKS cluster running k8s 1.21 - Falco is installed to it’s own falco namespace. The output is sometimes missing k8s and container information, for example this is one output:

{
    "output": "20:55:40.524063855: Error Package management process launched in container (user=root user_loginuid=-1 command=gem /usr/local/bin/gem contents fluent-plugin-elasticsearch container_id=1c626dd1ab1b container_name=<NA> image=<NA>:<NA>) k8s.ns=<NA> k8s.pod=<NA> container=1c626dd1ab1b k8s.ns=<NA> k8s.pod=<NA> container=1c626dd1ab1b",
    "priority": "Error",
    "rule": "Launch Package Management Process in Container",
    "time": "2021-09-09T20:55:40.524063855Z",
    "output_fields": {
        "container.id": "1c626dd1ab1b",
        "container.image.repository": null,
        "container.image.tag": null,
        "container.name": null,
        "evt.time": 1631220940524063855,
        "k8s.ns.name": null,
        "k8s.pod.name": null,
        "proc.cmdline": "gem /usr/local/bin/gem contents fluent-plugin-elasticsearch",
        "user.loginuid": -1,
        "user.name": "root"
    }
}

If I restart the falco pods, everything works again, and the above issue becomes

{
    "output": "08:26:23.338086075: Error Package management process launched in container (user=root user_loginuid=-1 command=gem /usr/local/bin/gem contents fluent-plugin-elasticsearch container_id=2ded43ee4a42 container_name=k8s_fluentd_fluentd-56qhz_kube-system_ac262b2d-df79-4f8b-93e2-e1d9df697177_0 image=<redacted>.amazonaws.com/fluentd:latest) k8s.ns=kube-system k8s.pod=fluentd-56qhz container=2ded43ee4a42 k8s.ns=kube-system k8s.pod=fluentd-56qhz container=2ded43ee4a42",
    "priority": "Error",
    "rule": "Launch Package Management Process in Container",
    "time": "2021-09-10T08:26:23.338086075Z",
    "output_fields": {
        "container.id": "2ded43ee4a42",
        "container.image.repository": "<redacted>.amazonaws.com/fluentd",
        "container.image.tag": "latest",
        "container.name": "k8s_fluentd_fluentd-56qhz_kube-system_ac262b2d-df79-4f8b-93e2-e1d9df697177_0",
        "evt.time": 1631262383338086075,
        "k8s.ns.name": "kube-system",
        "k8s.pod.name": "fluentd-56qhz",
        "proc.cmdline": "gem /usr/local/bin/gem contents fluent-plugin-elasticsearch",
        "user.loginuid": -1,
        "user.name": "root"
    }
}

There are currently 2 nodes in the cluster, and if I observe the output for both falco pods, I can’t see a pattern for this behaviour, such as one pod always showing no output - it’s seemingly random

How to reproduce it

Install falco to EKS cluster using helm: h -n falco install falco falcosecurity/falco --set falco.jsonOutput=true,falco.jsonIncludeOutputProperty=true

Expected behaviour

See all container/pod information in ever output

Screenshots

Environment

Falco version:

0.29.1

System info:

{
  "machine": "x86_64",
  "nodename": "falco-27cs6",
  "release": "5.4.129-63.229.amzn2.x86_64",
  "sysname": "Linux",
  "version": "#1 SMP Tue Jul 20 21:22:08 UTC 2021"
}

(This was run from execing into a running falco container)

Cloud provider or hardware configuration: AWS EKS
OS:

Kernel:

Installation method: Kubernetes (helm)

Additional context

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 18 (6 by maintainers)

Most upvoted comments

Have the same issue with Falco 0.30 installed via Helm 1.16.0. EKS 1.21 with Bottlerocket 1.1.4.

K8S information is always missing.

A complete log:

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.30.0, driver version=3aa7a83bf7b9e6229a3824e3fd1f4452d1e95cb4
* Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
* Trying to download a prebuilt eBPF probe from https://download.falco.org/driver/3aa7a83bf7b9e6229a3824e3fd1f4452d1e95cb4/falco_bottlerocket_5.10.35_1.o
curl: (22) The requested URL returned error: 404
* Trying to compile the eBPF probe (falco_bottlerocket_5.10.35_1.o)
Unable to find a prebuilt falco eBPF probe
scripts/Makefile.lib:8: 'always' is deprecated. Please use 'always-y' instead
In file included from /usr/src/falco-3aa7a83bf7b9e6229a3824e3fd1f4452d1e95cb4/bpf/probe.c:13:
In file included from ./include/linux/sched.h:14:
In file included from ./include/linux/pid.h:6:
In file included from ./include/linux/wait.h:9:
In file included from ./include/linux/spinlock.h:90:
In file included from ./arch/x86/include/asm/spinlock.h:27:
In file included from ./arch/x86/include/asm/qspinlock.h:111:
./include/asm-generic/qspinlock.h:68:55: warning: passing 'u32 *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types with different sign [-Wpointer-sign]
        return likely(atomic_try_cmpxchg_acquire(&lock->val, &val, _Q_LOCKED_VAL));
                                                             ^~~~
./include/linux/compiler.h:77:40: note: expanded from macro 'likely'
# define likely(x)      __builtin_expect(!!(x), 1)
                                            ^
./include/asm-generic/atomic-instrumented.h:703:46: note: passing argument to parameter 'old' here
atomic_try_cmpxchg_acquire(atomic_t *v, int *old, int new)
                                             ^
In file included from /usr/src/falco-3aa7a83bf7b9e6229a3824e3fd1f4452d1e95cb4/bpf/probe.c:13:
In file included from ./include/linux/sched.h:14:
In file included from ./include/linux/pid.h:6:
In file included from ./include/linux/wait.h:9:
In file included from ./include/linux/spinlock.h:90:
In file included from ./arch/x86/include/asm/spinlock.h:27:
In file included from ./arch/x86/include/asm/qspinlock.h:111:
./include/asm-generic/qspinlock.h:82:52: warning: passing 'u32 *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types with different sign [-Wpointer-sign]
        if (likely(atomic_try_cmpxchg_acquire(&lock->val, &val, _Q_LOCKED_VAL)))
                                                          ^~~~
./include/linux/compiler.h:77:40: note: expanded from macro 'likely'
# define likely(x)      __builtin_expect(!!(x), 1)
                                            ^
./include/asm-generic/atomic-instrumented.h:703:46: note: passing argument to parameter 'old' here
atomic_try_cmpxchg_acquire(atomic_t *v, int *old, int new)
                                             ^
In file included from /usr/src/falco-3aa7a83bf7b9e6229a3824e3fd1f4452d1e95cb4/bpf/probe.c:13:
In file included from ./include/linux/sched.h:14:
In file included from ./include/linux/pid.h:6:
In file included from ./include/linux/wait.h:9:
In file included from ./include/linux/spinlock.h:90:
In file included from ./arch/x86/include/asm/spinlock.h:43:
In file included from ./arch/x86/include/asm/qrwlock.h:6:
./include/asm-generic/qrwlock.h:65:56: warning: passing 'u32 *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types with different sign [-Wpointer-sign]
        return likely(atomic_try_cmpxchg_acquire(&lock->cnts, &cnts,
                                                              ^~~~~
./include/linux/compiler.h:77:40: note: expanded from macro 'likely'
# define likely(x)      __builtin_expect(!!(x), 1)
                                            ^
./include/asm-generic/atomic-instrumented.h:703:46: note: passing argument to parameter 'old' here
atomic_try_cmpxchg_acquire(atomic_t *v, int *old, int new)
                                             ^
In file included from /usr/src/falco-3aa7a83bf7b9e6229a3824e3fd1f4452d1e95cb4/bpf/probe.c:13:
In file included from ./include/linux/sched.h:14:
In file included from ./include/linux/pid.h:6:
In file included from ./include/linux/wait.h:9:
In file included from ./include/linux/spinlock.h:90:
In file included from ./arch/x86/include/asm/spinlock.h:43:
In file included from ./arch/x86/include/asm/qrwlock.h:6:
./include/asm-generic/qrwlock.h:92:53: warning: passing 'u32 *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types with different sign [-Wpointer-sign]
        if (likely(atomic_try_cmpxchg_acquire(&lock->cnts, &cnts, _QW_LOCKED)))
                                                           ^~~~~
./include/linux/compiler.h:77:40: note: expanded from macro 'likely'
# define likely(x)      __builtin_expect(!!(x), 1)
                                            ^
./include/asm-generic/atomic-instrumented.h:703:46: note: passing argument to parameter 'old' here
atomic_try_cmpxchg_acquire(atomic_t *v, int *old, int new)
                                             ^
4 warnings generated.
* eBPF probe located in /root/.falco/falco_bottlerocket_5.10.35_1.o
* Success: eBPF probe symlinked to /root/.falco/falco-bpf.o
Mon Nov  8 13:16:08 2021: Falco version 0.30.0 (driver version 3aa7a83bf7b9e6229a3824e3fd1f4452d1e95cb4)
Mon Nov  8 13:16:08 2021: Falco initialized with configuration file /etc/falco/falco.yaml
Mon Nov  8 13:16:08 2021: Configured rules filenames:
Mon Nov  8 13:16:08 2021:    /etc/falco/falco_rules.yaml
Mon Nov  8 13:16:08 2021:    /etc/falco/rules.d/custom_rules.yaml
Mon Nov  8 13:16:08 2021: Loading rules from file /etc/falco/falco_rules.yaml:
Mon Nov  8 13:16:08 2021: Loading rules from file /etc/falco/rules.d/custom_rules.yaml:
Mon Nov  8 13:16:09 2021: Setting metadata download max size to 100 MB
Mon Nov  8 13:16:09 2021: Setting metadata download chunk wait time to 1000 μs
Mon Nov  8 13:16:09 2021: Setting metadata download watch frequency to 1 seconds
{"output":"13:16:23.890609615: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=<NA> k8s.pod=<NA> container=f5eed781c4c5 shell=bash parent=runc cmdline=bash terminal=34816 container_id=f5eed781c4c5 image=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=f5eed781c4c5","priority":"Notice","rule":"Terminal shell in container","source":"syscall","tags":["container","mitre_execution","shell"],"time":"2021-11-08T13:16:23.890609615Z", "output_fields": {"container.id":"f5eed781c4c5","container.image.repository":null,"evt.time":1636377383890609615,"k8s.ns.name":null,"k8s.pod.name":null,"proc.cmdline":"bash","proc.name":"bash","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}
{"output":"13:16:23.988641684: Error File below / or /root opened for writing (user=root user_loginuid=-1 command=touch /root/.dirs parent=bash file=/root/.dirs program=touch container_id=f5eed781c4c5 image=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=f5eed781c4c5 k8s.ns=<NA> k8s.pod=<NA> container=f5eed781c4c5","priority":"Error","rule":"Write below root","source":"syscall","tags":["filesystem","mitre_persistence"],"time":"2021-11-08T13:16:23.988641684Z", "output_fields": {"container.id":"f5eed781c4c5","container.image.repository":null,"evt.time":1636377383988641684,"fd.name":"/root/.dirs","k8s.ns.name":null,"k8s.pod.name":null,"proc.cmdline":"touch /root/.dirs","proc.name":"touch","proc.pname":"bash","user.loginuid":-1,"user.name":"root"}}
{"output":"13:17:17.314025602: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=<NA> k8s.pod=<NA> container=f5eed781c4c5 shell=bash parent=runc cmdline=bash terminal=34816 container_id=f5eed781c4c5 image=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=f5eed781c4c5","priority":"Notice","rule":"Terminal shell in container","source":"syscall","tags":["container","mitre_execution","shell"],"time":"2021-11-08T13:17:17.314025602Z", "output_fields": {"container.id":"f5eed781c4c5","container.image.repository":null,"evt.time":1636377437314025602,"k8s.ns.name":null,"k8s.pod.name":null,"proc.cmdline":"bash","proc.name":"bash","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}

juris on Nov 8, 2021