falco: Not able to detect k8s metadata with containerd

Describe the bug

We switched our kubernetes clusters from docker to containerd. After we are not able to see kubernetes metadatas at all in falco events.

After (with docker): Screenshot 2021-03-08 at 17 11 52

Before (with containerd): Screenshot 2021-03-06 at 18 37 37

How to reproduce it

Install kubernetes cluster with manifests https://gist.github.com/zetaab/e134a6dbb80de296909a6d7ed34c3232

Expected behaviour

We expect that we could see kubernetes metadatas in events like it did work with docker.

Screenshots

Environment

Kubernetes cluster installed with kOps running in OpenStack / AWS

Falco version: 0.27.0
System info:

# falco --support | jq .system_info
{
  "machine": "x86_64",
  "nodename": "nodes-esptnl-rzkxpp",
  "release": "4.19.0-13-cloud-amd64",
  "sysname": "Linux",
  "version": "#1 SMP Debian 4.19.160-2 (2020-11-28)"
}

Cloud provider or hardware configuration: openstack / aws
OS: debian buster
Kernel: 4.19.0-13-cloud-amd64
Installation method: kubernetes

Additional context

falco kubernetes installation manifests https://gist.github.com/zetaab/e134a6dbb80de296909a6d7ed34c3232

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 25 (3 by maintainers)

Most upvoted comments

not working containerd config:

root@nodes-helpa-hgah1q:~# cat /etc/containerd/config-kops.toml
version = 2

[plugins]

  [plugins."io.containerd.grpc.v1.cri"]

    [plugins."io.containerd.grpc.v1.cri".containerd]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"

working

root@nodes-helpa-zj2t5t:/home/debian# cat /etc/containerd/config-kops.toml
version = 2

[plugins]

  [plugins."io.containerd.grpc.v1.cri"]

    [plugins."io.containerd.grpc.v1.cri".containerd]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = true

so our way to fix this is to roll all machines in clusters.

zetaab on Mar 14, 2021

For others that might have the same issue, I ran into this problem today, and it ended up being a misconfigured containerd.socket when using k3s.

Following the guidance from @holyspectral in this post (thank you), the grpcurl was returning connection refused. The problem was the path to the socket. After figuring out the correct path I could then start falco with the correct settings and everything worked as it should. Here it the command I am using now:

helm install falco \
    --create-namespace \
    --namespace falco \
    --set containerd.enabled=true \
    --set containerd.socket=/run/k3s/containerd/containerd.sock \
falcosecurity/falco

pmusa on Oct 11, 2021

@leogr I have falco built from master branch with sysdig dev branch. The ebpf program is built from source and kernel header. They’re installed via helm.

holyspectral on Mar 11, 2021

# ./grpcurl -plaintext -proto cri.proto -unix /host/run/containerd/containerd.sock runtime.v1alpha2.RuntimeService/Version
{
  "version": "0.1.0",
  "runtimeName": "containerd",
  "runtimeVersion": "v1.4.3",
  "runtimeApiVersion": "v1alpha2"
}

ListContainers will return array of containers

one example:

    {
      "id": "2f0fc4d462d7b55a7684c21a3c781c385d0e90c323dc71610d873a323a920f20",
      "podSandboxId": "8fb7f10600b27691fe4d9c644ec40e26d1ce1a6cabf7212188ffccb8c62c14e3",
      "metadata": {
        "name": "postgres"
      },
      "image": {
        "image": "sha256:629b479e767ba8b78e6dbd366bf0235abb3f4d2d8ca4bcca82df56543e322eb4"
      },
      "imageRef": "sha256:629b479e767ba8b78e6dbd366bf0235abb3f4d2d8ca4bcca82df56543e322eb4",
      "state": "CONTAINER_RUNNING",
      "createdAt": "1614670197868951320",
      "labels": {
        "io.kubernetes.container.name": "postgres",
        "io.kubernetes.pod.name": "dapi-sre-sandbox-2",
        "io.kubernetes.pod.namespace": "dapi",
        "io.kubernetes.pod.uid": "7ef0df70-d762-49bc-a498-59ed4fb0ba2b"
      },
      "annotations": {
        "io.kubernetes.container.hash": "6a88411c",
        "io.kubernetes.container.ports": "[{\"containerPort\":8008,\"protocol\":\"TCP\"},{\"containerPort\":5432,\"protocol\":\"TCP\"},{\"containerPort\":8080,\"protocol\":\"TCP\"}]",
        "io.kubernetes.container.restartCount": "0",
        "io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
        "io.kubernetes.container.terminationMessagePolicy": "File",
        "io.kubernetes.pod.terminationGracePeriod": "300"
      }
    },

btw executed this inside running falco container I also tried it locally in node. The result is same in both (when running in host I removed /host from start of the containerd sock path)

zetaab on Mar 11, 2021

We are missing the metadata all the time.

As you can see from the gist provided in the issue, we have already check & tried the common problems. HOST_ROOT env variable is not an issue and neither is the async fetching, as toggling it on & off does not make a difference.

Are there any other details that you might need? We are running containerd v1.4.3.

ottosulin on Mar 11, 2021