fluent-bit: Memory leak in kubernetes filter plugin when using K8S-Logging.Exclude: On in fluent-bit version 1.9.0 and 1.8

Bug Report

Describe the bug

There is a memory leak in kubernetes filter when using K8S-Logging.Exclude On to block logs from pods.

To Reproduce Run fluent-bit version 1.9.0, 1.8.13, 1.8.12 or 1.8.4 with the kubernetes filter using K8S-Logging.Exclude On and some pods annotated with fluentbit.io/exclude: "true"

Current behavior

when the K8S-Logging.Exclude option is turned On, memory leakage occurs. It remains constant when the option is turned Off.

Expected behavior

Memory usage stays constant with constant load.

Screenshots

The memory usage start increasing with K8S-Logging.Exclude option turned On. It remains constant when it is turned Off.

Your Environment

• Version used: fluent-bit v1.9.0, v1.8.13, v1.8.12, v1.8.4

• Configuration: A simpler version of the configuration we used will include a systemd input plugin, kubernetes filter plugin and forward output plugin.

• Running on Kubernetes v1.21 on top of linux/amd64 in a single node cluster.

Additional context

similar issue https://github.com/fluent/fluent-bit/issues/1068 for reference.