charts: Helm Chart; Auditing not working. "the server could not find the requested resource"

Describe the bug

When installing Falco through the helm chart, this issue falcosecurity/falco#1026, that relates to a wrong setting in the Auditsink, still persists. After setting this to the correct format, as described in this issue, my Kubernetes API log file is full with Error in audit plugin 'dynamic_webhook' affecting 1 audit events: the server could not find the requested resource errors. Tried generating some Kubernetes audits, that should trigger an alert for Falco as described on the Falco documentation site, but no alert is given. This probably indicates that Falco is not receiving any audit logs.

How to reproduce it

Add the following to the Kubernetes API server:

• –audit-dynamic-configuration
• –feature-gates=DynamicAuditing=true
• –runtime-config=auditregistration.k8s.io/v1alpha1=true

Set “auditLog”, and “dynamicBackend” to true in the values.yaml, provided by the Falco Helm chart.

Install Falco with the Helm Chart with the command: helm install Falco -f values.yaml stable/falco. Used Helm 3.2.1, so the original commands on the Falco Chart Github site won’t work anymore.

Expected behaviour

Audit logs from the Kubernetes API server getting received and inspected by Falco.

Screenshots

2020-05-11T11:05:24.005466424Z AUDIT: id="0b967b34-9750-4dcd-905c-cacf392c16c7" stage="ResponseComplete" ip="xx.xx.xx.xx" method="get" user="system:kube-controller-manager" groups="\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s" response="200"
E0511 11:16:46.538374       1 metrics.go:109] Error in audit plugin 'dynamic_webhook' affecting 1 audit events: the server could not find the requested resource
Impacted events:
2020-05-11T11:08:12.927495363Z AUDIT: id="d031c5b4-101e-4ba9-964f-fb8cc0b9b402" stage="ResponseComplete" ip="xx.xx.xx.xx" method="update" user="system:kube-controller-manager" groups="\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=10s" response="200"
E0511 11:16:46.581997       1 metrics.go:109] Error in audit plugin 'dynamic_webhook' affecting 1 audit events: the server could not find the requested resource
Impacted events:
2020-05-11T11:02:22.342627937Z AUDIT: id="ec410dc7-8173-4528-aef6-d66a753524df" stage="ResponseStarted" ip="xx.xx.xx.xx" method="watch" user="system:node:workernode1" groups="\"system:nodes\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/secrets?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dfalco-token-6z6x6&resourceVersion=33914&timeout=8m14s&timeoutSeconds=494&watch=true" response="200"
E0511 11:16:46.677243       1 metrics.go:109] Error in audit plugin 'dynamic_webhook' affecting 1 audit events: the server could not find the requested resource
Impacted events:
2020-05-11T11:15:01.198249466Z AUDIT: id="0046feb5-dea3-4361-b9e2-3472e01537e9" stage="ResponseComplete" ip="xx.xx.xx.xx" method="get" user="system:kube-controller-manager" groups="\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s" response="200"
E0511 11:16:46.880651       1 metrics.go:109] Error in audit plugin 'dynamic_webhook' affecting 1 audit events: the server could not find the requested resource
Impacted events:
2020-05-11T11:01:40.478150167Z AUDIT: id="12f28d4c-4e36-40a5-b3e0-faabb666b17d" stage="ResponseComplete" ip="xx.xx.xx.xx" method="get" user="system:serviceaccount:kube-system:generic-garbage-collector" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/apiextensions.k8s.io/v1beta1?timeout=32s" response="200"
E0511 11:16:46.903637       1 metrics.go:109] Error in audit plugin 'dynamic_webhook' affecting 1 audit events: the server could not find the requested resource
Impacted events:
2020-05-11T11:12:18.187986702Z AUDIT: id="e53e4dad-9a1d-49d3-95de-f2e26de39259" stage="ResponseComplete" ip="xx.xx.xx.xx" method="update" user="system:kube-scheduler" groups="\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s" response="200"
E0511 11:16:46.908830       1 metrics.go:109] Error in audit plugin 'dynamic_webhook' affecting 1 audit events: the server could not find the requested resource
Impacted events:
2020-05-11T11:02:00.904053739Z AUDIT: id="e0b80922-c48c-4cbf-86ac-a5e545a5e2dc" stage="ResponseComplete" ip="xx.xx.xx.xx" method="get" user="system:kube-controller-manager" groups="\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=10s" response="200"
E0511 11:16:46.959913       1 metrics.go:109] Error in audit plugin 'dynamic_webhook' affecting 1 audit events: the server could not find the requested resource
Impacted events:
2020-05-11T11:05:43.536143494Z AUDIT: id="855dcc7e-0fad-4d3f-bb8e-e7035adf48e4" stage="ResponseStarted" ip="xx.xx.xx.xx" method="watch" user="system:kube-controller-manager" groups="\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/rbac.authorization.k8s.io/v1/clusterroles?allowWatchBookmarks=true&resourceVersion=33914&timeout=9m1s&timeoutSeconds=541&watch=true" response="200"
E0511 11:16:47.060695       1 metrics.go:109] Error in audit plugin 'dynamic_webhook' affecting 1 audit events: the server could not find the requested resource
Impacted events:
2020-05-11T11:08:30.947512433Z AUDIT: id="4c2b9bd7-c162-496b-af08-50e3744a0c5c" stage="ResponseComplete" ip="xx.xx.xx.xx" method="get" user="system:kube-scheduler" groups="\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=10s" response="200"

Environment

• Falco version: 0.22.1

• System info: { “machine”: “x86_64”, “nodename”: “falco-b8kjr”, “release”: “4.18.0-147.8.1.el8_1.x86_64”, “sysname”: “Linux”, “version”: “#1 SMP Thu Apr 9 13:49:54 UTC 2020” }

• Cloud provider or hardware configuration: Kubernetes 1.18.2
• OS: NAME=“CentOS Linux” VERSION=“8 (Core)”

• Kernel: 4.18.0-147.8.1.el8_1.x86_64

• Installation method: Helm Chart

Additional context Tried installing Falco as a host-based installation via the script on the Falco documention site. By using this method and configuring the Kubernetes API server, Falco works as expected and no issues appear in the Kubernetes API log.

Added this issue couple of days ago. When I looked at it agian, it was moved to the contrib section? Why? This is not a contribute report.