fluent-bit: Renaming keys not working

Bug Report

Describe the bug Renaming keys with modify filter plugin is not working. The new keys renamed are not showing.

To Reproduce apply these filters (kubernetes environment):

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log           On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off

[FILTER]
    Name modify
    Match *
    Rename kubernetes.namespace_name company
    Rename kubernetes.pod_name pod
    Remove_wildcard kubernetes

Expected behavior logs without kubernetes keys and with the new keys company and pod

Your Environment

Version used: 1.1.0
Environment name and version (e.g. Kubernetes? What version?): 1.11.9
Filters and plugins: kubernetes, modify

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 9
Comments: 19 (9 by maintainers)

Most upvoted comments

I’m also getting this behaviour. To elaborate a bit: The old key in my case is still there, it just does not get renamed as specified.

Version used: 1.3.7 Environment: Kubernetes 1.18.0 Filters and plugins: kubernetes, modify

windrammer on Apr 27, 2020

@q2dg @carlosrmendes I believe it is the same issue as: https://github.com/fluent/fluent-bit/issues/2152

The documentation for the “modify” filter uses “Mem.free” which makes it seem like this should be possible. But “Mem.free” is the key. So not nested (at least I think).

So in your case, you may have to use the “nested” filter to lift, add a prefix, then copy, then nest and remove the prefix.

For example @carlosrmendes:

    [FILTER]
        Name nest
        Match kube.*
        Operation lift
        Nested_under kubernetes
        Add_prefix kubernetes_
        
    [FILTER]
        Name modify
        Match  kube.*
        Rename kubernetes_namespace_name company
        Rename kubernetes_pod_name pod

Now it is at the root and you can modify it. Or copy it. The issue is now everything is at the root. Which is why after copying, you may want to nest back depending on what you are doing

TBH seeing all these issues get ignored makes me a bit worried about fluent bit and its future.

corey-aloia on Mar 8, 2023

Closing issues is not a good way to deal with them

q2dg on Jun 7, 2023

This is issue is important enough to not being staled, I think

q2dg on Apr 12, 2022