falco: version 0.33.1, chart 2.4.3 EKS: k8saudit-eks plugin does not load

Here’s my whole values.yaml I’m using for my demos:

tty: true
kubernetes: false

customRules:
  override-k8saudit.yaml: |-
    - list: allowed_k8s_users
      append: true
      items: [eks:cloud-controller-manager, eks:vpc-resource-controller, eks:az-poller]
    - macro: live_endpoint
      append: true
      condition: or ka.uri="/readyz?exclude=kms-provider-0" or ka.uri="/livez?exclude=kms-provider-0"

falco:
  rules_file:
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d
  plugins:
    - name: k8saudit-eks
      library_path: libk8saudit-eks.so
      init_config:
        region: "us-east-1"
        # profile: "default"
        shift: 10
        polling_interval: 10
        use_async: false
        buffer_size: 500
      open_params: "falco-demo"
    - name: json
      library_path: libjson.so
      init_config: ""
  load_plugins: [k8saudit-eks, json]
  json_output: true
  json_include_output_property: true
  json_include_tags_property: true
  http_output:
    enabled: true
    url: "http://falco-falcosidekick.falco:2801"

driver:
  enabled: false
collectors:
  enabled: false

controller:
  kind: deployment
  deployment:
    replicas: 1

falcoctl:
  indexes:
  - name: falcosecurity
    url: https://falcosecurity.github.io/falcoctl/index.yaml
  artifact:
    install:
      enabled: true
    follow:
      enabled: true
  config:
    artifact:
      allowedTypes:
        - plugin
        - rulesfile
      install:
        resolveDeps: false
        refs: [k8saudit-rules:0.5, k8saudit-eks:0.1, json:0]
      follow:
        refs: [k8saudit-rules:0.5]

I’ve a different daemonset for falco + falcosidekick + falcosidekick-ui, this is why I set also the json_* and http_output values.

I think your missing config is :

falco:
  rules_file:
    - /etc/falco/k8s_audit_rules.yaml

I will hijack this issue and use this comment to document me trying to get it to work.

I download the plugin and built my own image. For now I am using the stable version of the plugin. I get it from here:

• https://download.falco.org/plugins/stable/k8saudit-eks-0.1.0-linux-x86_64.tar.gz

Building the image in the following temporary repo:

• https://github.com/trallnag/falco-with-plugins

I got the plugin to load!

Fri Jan 13 13:55:39 2023: Falco version: 0.33.1 (x86_64)
Fri Jan 13 13:55:39 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri Jan 13 13:55:39 2023: Loading plugin 'k8saudit-eks' from file /usr/share/falco/plugins/libk8saudit-eks.so
Fri Jan 13 13:55:39 2023: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Fri Jan 13 13:55:39 2023: Loading rules from file /etc/falco/falco_rules.yaml
Fri Jan 13 13:55:39 2023: Loading rules from file /etc/falco/falco_rules.local.yaml
Fri Jan 13 13:55:40 2023: Starting health webserver with threadiness 4, listening on port 8765
Fri Jan 13 13:55:40 2023: Enabled event sources: k8s_audit
Fri Jan 13 13:55:40 2023: Opening capture with plugin 'k8saudit-eks'
Error: RequestError: send request failed
caused by: Post "https://logs.us-east-1.amazonaws.com/": x509: certificate signed by unknown authority
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:

@rtalipov, maybe there is something wrong with your config?

Here is are my overrides:

image:
  pullPolicy: Always
  registry: ghcr.io/trallnag
  repository: falco-with-plugins
  tag: master

controller:
  kind: deployment

driver:
  enabled: false

collectors:
  enabled: false

auditLog:
  enabled: true

falco:
  plugins:
    - name: k8saudit-eks
      library_path: libk8saudit-eks.so
      init_config:
        region: "us-east-1"
        profile: "default"
        shift: 10
        polling_interval: 10
        use_async: false
        buffer_size: 500
      open_params: "c360-dev"
    - name: json
      library_path: libjson.so
      init_config: ""
  load_plugins: [k8saudit-eks, json]

Next I will add the certificates to the custom image. There is already a PR open for that by @Issif, but I don’t want to wait.

Installing certs works.

Error: AccessDeniedException: User: arn:aws:sts::023551243526:assumed-role/c360-dev-eks-ng-apps/i-0efaa0f48d3cea5de is not authorized to perform: logs:FilterLogEvents on resource: arn:aws:logs:us-east-1:023551243526:log-group:/aws/eks/c360-dev/cluster:log-stream

Next I will configure permissions.

These seem to be enough:

{
    "Statement": [
        {
            "Action": [
                "logs:Describe*",
                "logs:FilterLogEvents",
                "logs:Get*",
                "logs:List*"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:logs:eu-central-1:023551243526:log-group:/aws/eks/CLUSTER_NAME/cluster:*"
            ],
            "Sid": ""
        }
    ],
    "Version": "2012-10-17"
}

I have also overridden the included k8s audit rules because the images don’t contain the latest versions.

Still not working. Now it is complaining about plugin requirements:

Fri Jan 13 21:50:51 2023: Falco version: 0.33.1-105+c1985a7 (x86_64)
Fri Jan 13 21:50:51 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri Jan 13 21:50:51 2023: Loading plugin 'k8saudit-eks' from file /usr/share/falco/plugins/libk8saudit-eks.so
Fri Jan 13 21:50:51 2023: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Fri Jan 13 21:50:51 2023: Loading rules from file /etc/falco/falco_rules.yaml
Fri Jan 13 21:50:51 2023: Loading rules from file /etc/falco/k8s_audit_rules.yaml
Error: Plugin requirement not satisfied, must load one of: k8saudit (>= 0.1.0)

But I definitely have placed 0.1.0 version of plugin and the rules yaml looks right.

- required_engine_version: 15
- required_plugin_versions:
  - name: k8saudit-eks
    version: 0.1.0

- rule: Dummy rule
  desc: >
        Dummy rule
  condition: >
        ka.verb in (get,create,delete,update)
  output: user=%ka.user.name verb=%ka.verb target=%ka.target.name target.namespace=%ka.target.namespace resource=%ka.target.resource
  priority: WARNING
  source: k8s_audit
  tags: [k8s]

I finally figured it out. The chart is bringing it’s own rules and it is overwriting the rules within the image.

You can use any value you want for “name” (alphanumeric chars and - work definitely). It is just used as an identifier for the respective exception.

Here’s the minimal policy you need: https://github.com/falcosecurity/plugins/tree/master/plugins/k8saudit-eks#aws-iam-policy-permissions

Here’s my whole values.yaml I’m using for my demos:

tty: true
kubernetes: false

customRules:
  override-k8saudit.yaml: |-
    - list: allowed_k8s_users
      append: true
      items: [eks:cloud-controller-manager, eks:vpc-resource-controller, eks:az-poller]
    - macro: live_endpoint
      append: true
      condition: or ka.uri="/readyz?exclude=kms-provider-0" or ka.uri="/livez?exclude=kms-provider-0"

falco:
  rules_file:
    - /etc/falco/k8s_audit_rules.yaml
    - /etc/falco/rules.d
  plugins:
    - name: k8saudit-eks
      library_path: libk8saudit-eks.so
      init_config:
        region: "us-east-1"
        # profile: "default"
        shift: 10
        polling_interval: 10
        use_async: false
        buffer_size: 500
      open_params: "falco-demo"
    - name: json
      library_path: libjson.so
      init_config: ""
  load_plugins: [k8saudit-eks, json]
  json_output: true
  json_include_output_property: true
  json_include_tags_property: true
  http_output:
    enabled: true
    url: "http://falco-falcosidekick.falco:2801"

driver:
  enabled: false
collectors:
  enabled: false

controller:
  kind: deployment
  deployment:
    replicas: 1

falcoctl:
  indexes:
  - name: falcosecurity
    url: https://falcosecurity.github.io/falcoctl/index.yaml
  artifact:
    install:
      enabled: true
    follow:
      enabled: true
  config:
    artifact:
      allowedTypes:
        - plugin
        - rulesfile
      install:
        resolveDeps: false
        refs: [k8saudit-rules:0.5, k8saudit-eks:0.1, json:0]
      follow:
        refs: [k8saudit-rules:0.5]

I’ve a different daemonset for falco + falcosidekick + falcosidekick-ui, this is why I set also the json_* and http_output values.

I think your missing config is :

falco:
  rules_file:
    - /etc/falco/k8s_audit_rules.yaml

@Issif, yes that was the case, now it’s working, thanks a lot!