fluent-bit: Filter nest/lift is broken on Windows in 2.1.10 and 2.2.0 =>logs corrupted

Bug Report

Describe the bug After upgrading from Fluent Bit 2.1.9 to 2.1.10 the nest/lift filter is no longer working and corrupts the logs

My case are Windows pods in k8s

Fluent Bit reads the log files and adds Kubernetes information. Then it nest/lift the Kubernetes fields

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        DNS_Retries         10
        DNS_Wait_Time       30
        Merge_Log           On
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Labels              On
        Annotations         Off
    [FILTER]
        Name                nest
        Match               kube.*
        Operation           lift
        Nested_under        kubernetes
        Add_prefix          _kubernetes.

The result in cr.fluentbit.io/fluent/fluent-bit:windows-2019-2.1.10 would look something like this:

{"time"=>"2023-11-06T14:04:33.8847011Z", "stream"=>"stdout", "_p"=>"F", "RequestHttpVerb"=>"GET", "RequestUri"=>"http://art.staging.firmglobal.net/identity/.well-known/openid-configuration", "ResponseStatusCode"=>"200", "TimeTakenMicros"=>"12622", "Service"=>"Confirmit.Identity", "ServiceVersion"=>"2.24.1753", "CorrelationId"=>"23015ddcfebe4f7e9fc2d180318e0adb", "InitiatingService"=>"Confirmit.Identity", "EventTime"=>"2023-11-06T14:04:33.863+00", "Severity"=>"INFO", "Logger"=>"Confirmit.Logging.Web.AccessLogger", "RequestHeaders"=>"Host={art.staging.firmglobal.net}", "_kubernetes.pod_name�identity3-api-7c5c56447f-hb6rv�|_kubernetes.namespace_n"=>97, 109=>101, "horizons"=>"_kubernetes.pod_id�$09cc3c99-a4ac-4f54-a884-02bf196b5069�<_k", 117=>98, 101=>114, 110=>101, 116=>101, 115=>46, 108=>97}

As you can see there are some weird characters in the log after doing nest/lift. And also some weird 117=>98, 101=>114, 110=>101, 116=>101, 115=>46, 108=>97.

All this works fine when I revert the container image to cr.fluentbit.io/fluent/fluent-bit:windows-2019-2.1.9

This is how the output looks in 2.1.9

{"time"=>"2023-11-06T14:34:58.7061335Z", "stream"=>"stdout", "_p"=>"F", "RequestHttpVerb"=>"GET", "RequestUri"=>"http://10.241.74.235/identity/.well-known/openid-configuration", "ResponseStatusCode"=>"200", "TimeTakenMicros"=>"34615", "Service"=>"Confirmit.Identity", "ServiceVersion"=>"2.24.1753", "CorrelationId"=>"b3f7e7854ae041ce9bb3baaeeeade543", "InitiatingService"=>"Confirmit.Identity", "EventTime"=>"2023-11-06T14:34:58.684+00", "Severity"=>"INFO", "Logger"=>"Confirmit.Logging.Web.AccessLogger", "RequestHeaders"=>"Connection={close}, Accept={*/*}, Host={10.241.74.235:80}", "_kubernetes.pod_name"=>"identity3-api-7c5c56447f-jkhhf", "_kubernetes.namespace_name"=>"horizons", "_kubernetes.pod_id"=>"b56eef39-ac81-4efc-959f-bea7dd3b0193", "_kubernetes.labels"=>{"app"=>"identity3-api", "app.confirmit.com/template"=>"iis-app", "app.confirmit.com/template-version"=>"11.2.0", "app.kubernetes.io/instance"=>"horizons", "app.kubernetes.io/managed-by"=>"Helm", "app.kubernetes.io/name"=>"identity3-api", "app.kubernetes.io/version"=>"2.24.1753", "helm.sh/chart"=>"identity3-api-2.24.1753", "pod-template-hash"=>"7c5c56447f", "version"=>"2.24.1753"}, "_kubernetes.host"=>"akswinu7f0000t2", "_kubernetes.container_name"=>"identity3-api", "_kubernetes.docker_id"=>"0a4d2404db4070675163eac1bf4cc7062d82ffd54c71fadbc14045436c5cee24", "_kubernetes.container_hash"=>"confirmithorizonsdev.azurecr.io/confirmit/identity3-api@sha256:0b6349dab35536371cc88385bacc33cfb1138ef9ddc99729bc6efb2a71870f9f", "_kubernetes.container_image"=>"confirmithorizonsdev.azurecr.io/confirmit/identity3-api:2.24.1753"}

For Linux containers nest/lift works as before after the upgrade. But I need it working for for Windows pods also

Let me know if you need more information to reproduce the issue

About this issue

  • Original URL
  • State: closed
  • Created 8 months ago
  • Reactions: 11
  • Comments: 32 (6 by maintainers)

Most upvoted comments

Confirmed fixed in latest version

It is in the review queue, once merged it will go into the next release.