fluent-bit: Multiline Filter example does not work with Forward input

Bug Report

Describe the bug

I took the multiline example here and turned it into an example that works with Forward input: https://docs.fluentbit.io/manual/pipeline/filters/multiline-stacktrace

However, that didn’t work. I am not able to get the filter working and actually concatenating multilines with the forward input. Which is strange, since there shouldn’t be a difference between my forward example and the tail example from the POV of the filter.

To Reproduce

First, create a container image that will output the log file from the example to stdout:

FROM public.ecr.aws/amazonlinux/amazonlinux:latest
ADD test.log /test.log

RUN yum upgrade -y && yum install -y python3 pip3

RUN pip3 install boto3

WORKDIR /usr/local/bin

COPY main.py .

CMD ["python3", "main.py"]

You could just cat the file… but I went for a python script that prints it line by line:

file1 = open('test.log', 'r')
Lines = file1.readlines()
 
# Strips the newline character because print will add it back
for line in Lines:
    print(line.rstrip())

test.log is the file from the example in the docs.

You can run this container with the fluentd docker log driver:

docker run -it --log-driver fluentd app

And then capture the logs with this configuration:

[SERVICE]
    flush                 1
    grace                 1
    log_level             info
    parsers_file          parsers_multiline.conf

# [INPUT]
#     name                  tail
#     path                  /fluent-bit/etc/test.log
#     read_from_head        true
[INPUT]
    Name forward
    Listen 0.0.0.0
    Port 24224

[FILTER]
    Name    modify
    Match   *
    Remove_wildcard container_name
    Remove_wildcard source
    Remove_wildcard container_id


[FILTER]
    name                  multiline
    match                 *
    multiline.key_content log
    multiline.parser      go, multiline-regex-test



[OUTPUT]
    name                  stdout
    match                 *
    Format                json_lines
    json_date_key         false

The parsers file is the same as the one from the example.

• Steps to reproduce the problem:

Expected behavior

Multiline example should work with forward input.

Screenshots

Your Environment

• Version used: 1.8.7
• Environment name and version (e.g. Kubernetes? What version?): My MacBook

Additional context

Amazon ECS FireLens uses the Fluentd Docker Log Driver and forward input for logs, and so this issue has impacted many AWS customers: https://github.com/aws/aws-for-fluent-bit/issues/100