fluent-bit: `modify` filter does not respect nested keys

Still a valid issue, commenting so this doesn’t get closed.

Not sure how you would support this perfectly without a new parameter with a json path. I’ve used nest like this, hopefully this helps people.

    [FILTER]
        Name                nest
        Match               kube.*
        Operation           lift
        Nested_under        kubernetes
        Add_prefix          temp_l1_

    [FILTER]
        Name                nest
        Match               kube.*
        Operation           lift
        Nested_under        temp_l1_labels
        Add_prefix          temp_l2_

    [FILTER]
        Name                modify
        Match               kube.*
        Rename              temp_l2_label1 label1
        Rename              temp_l2_label2 label2

    [FILTER]
        Name                nest
        Match               kube.*
        Operation           nest
        Wildcard            temp_l2_*
        Nest_under          temp_l1_labels
        Remove_prefix       temp_l2_

    [FILTER]
        Name                nest
        Match               kube.*
        Operation           nest
        Wildcard            temp_l1_*
        Nest_under          kubernetes
        Remove_prefix       temp_l1_

But you can simply turn it off

[FILTER]
    Name                kubernetes
    Labels              Off
    Annotations         Off

Still a problem. I’d like to remove some kubernetes.examplekey but the filter doesn’t respect nested keys. record_modifier has the same problem

function remove_k8s_labels_annotations(tag, timestamp, record)
      record['kubernetes']['labels'] = nil
      record['kubernetes']['annotations'] = nil

This issue is still valid, why isn’t this issue not labeled as a bug?

I have the same issue ! i running fluent-bit from container image 1.8.6 latest

Input

{
  "method":"POST",
  "remoteAddr":"1.2.3.4",
  "user":{"group": ["a", "b", "c"]}
}

    [FILTER]
        Name    record_modifier
        Match   app.log
        Remove_key  remoteAddr
        Remove_key  user.group

Remove the nested key user.group not work

This is still an issue. It’s quite hard to maintain proper lua scripts for this. Would be nice to have native config to support kubernetes metadata manipulation.

Anyone found a way out? Any relation with https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/record-accessor?

Still a problem. I’d like to remove some kubernetes.examplekey but the filter doesn’t respect nested keys. record_modifier has the same problem

I encountered a similar issue and i managed to resolve it using a super simple lua script, hope it helps:

function remove_k8s_labels_annotations(tag, timestamp, record)
      record['kubernetes']['labels'] = nil
      record['kubernetes']['annotations'] = nil

      return 2, timestamp, record
    end

and you use it like this:

[FILTER]
        Name    lua
        Match   kube.*
        script  functions.lua
        call    remove_k8s_labels_annotations

Yea that might work for some people. I found the easiest was to set the kubernetes plugin filter to annotations off and lables off See docs for more details: https://docs.fluentbit.io/manual/pipeline/filters/kubernetes

Still a problem. I’d like to remove some kubernetes.examplekey but the filter doesn’t respect nested keys. record_modifier has the same problem

I encountered a similar issue and i managed to resolve it using a super simple lua script, hope it helps:

function remove_k8s_labels_annotations(tag, timestamp, record)
      record['kubernetes']['labels'] = nil
      record['kubernetes']['annotations'] = nil

      return 2, timestamp, record
    end

and you use it like this:

[FILTER]
        Name    lua
        Match   kube.*
        script  functions.lua
        call    remove_k8s_labels_annotations

Still a problem. I’d like to remove some kubernetes.examplekey but the filter doesn’t respect nested keys. record_modifier has the same problem

Hello, I ended up there trying to clean most Kubernetes keys while still collecting all of them for a separate output. My version is 2.2.1 (using debug Docker image) and the following config still don’t work:

[INPUT]
    Name               tail
    Path               /var/log/containers/*.log
    Parser             docker
    Tag                kube.*
    Mem_Buf_Limit      5MB
    Skip_Long_Lines    On
    Rotate_Wait        30
    Refresh_Interval   10
[FILTER]
    Name                  kubernetes
    Match                 kube.*
    Kube_URL              https://kubernetes.default.svc:443
    Kube_CA_File          /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File       /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix       kube.var.log.containers.
    Merge_Log             Off
    Merge_Log_Key         log_processed
    K8S-Logging.Parser    On
    K8S-Logging.Exclude   On
[FILTER]
    Name            record_modifier
    Match           *
    Allowlist_key   log
    Allowlist_key   kubernetes.pod_name
    Allowlist_key   kubernetes.container_image
    Allowlist_key   kubernetes.host
[OUTPUT]
    Name    file
    Match   *
    Path    /

I’m trying to keep log, kubernetes.pod_name, kubernetes.container_image and kubernetes.host, but the only key remaining in the end is log 😢 I think I will have to use the Lua workaround, but it’s really sad for such a basic task…

Hi friends,

for anyone else who’s bumping into this, I came up with the following solution using the lua filter:

      [FILTER]
          name lua
          match k8s_events
          call drop_managed_fields
          code function drop_managed_fields(tag, timestamp, record) record["metadata"]["managedFields"] = nil return 1, timestamp, record end

Inline the Lua code to have everything in one place, unfortunately that hurts readability a bit. The Lua code is as follows:

function drop_managed_fields(tag, timestamp, record)
    record["metadata"]["managedFields"] = nil
    return 1, timestamp, record end

(to delete an item from a Lua table, one has to set the value to nil: https://stackoverflow.com/questions/1758991/how-to-remove-a-lua-table-entry-by-its-key )

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

This problem is still actual not only for the modify plugin but GELF output plugin also. I cannot set a nested key for Gelf_Host_Key parameter.

The issue still exist in 1.7 😦

Can confirm this still does not work on v1.5.2. What I’m doing to work around, for now, is using the nest filter to lift records up and then process with rename. This isn’t optimal, though, as I’d ideally only like to lift 1 or 2 keys rather than the whole nested structure.