aws-for-fluent-bit: S3 output key formatting inconsistent rendering of `$TAG[n]`

Bug Report

Describe the bug S3 output has a key_format property which allows for using $TAG, or $TAG[n], (along with tag_delimiters) to create a path for objects based on tag properties. Sometimes using $TAG[n] in the path works fine, other times you get $TAG[n-1] and still other times you get full.tag.output.without.parsing[n] rendered.

To Reproduce

use the s3 output plugin
have logs with a tag like my.awesome.experimental.tag
set the tag_delimiter to .
set a key_format to something like /$TAG[2]/some_value=$TAG[3]/year=%Y/month=%m/day=%d/%H%M%S-$UUID.gz
see that sometimes it works perfectly. sometimes it treats tag[2] as tag[3] and sometimes it doesn’t parse the tag at all so the path results in /my.awesome.experimental.tag[3]/ all without changing the fluent-bit version, the app version, the tags, or anything else.

Expected behavior given my.awesome.experimental.tag, $TAG[2] should always parse to experimental and the s3 key path would then always render as /experimental/some_value=tag/

Your Environment aws-for-fluent-bit 2.25.1, fluentbit 1.9.3, Fargate and Firelens.

duplicate of https://github.com/fluent/fluent-bit/issues/5662 since we’re using aws-for-fluent-bit, wanted to raise awareness here as well.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 27 (13 by maintainers)

Most upvoted comments

here’s a few examples from my configuration, highlighting that I have numerous outputs and many different sub-indexes used in different outputs.

[OUTPUT]
    Match                       applicationtwo.logger.*
    Name                        s3
    Retry_Limit                 False
    bucket                      ${ATHENA_LOG_BUCKET}
    s3_key_format               /${SERVICE_NAME}/${ENVIRONMENT}/$TAG[3]/client_id=$TAG[4]/year=%Y/month=%m/day=%d/%H%M%S-$UUID.gz
    s3_key_format_tag_delimiters .
    compression                 gzip
    total_file_size             1M
    upload_timeout              30s
    use_put_object              On
    auto_retry_requests         true
    net.dns.mode                TCP
[OUTPUT]
    Match                       applicationone.*
    Name                        s3
    Retry_Limit                 False
    bucket                      ${ATHENA_LOG_BUCKET}
    s3_key_format               /${SERVICE_NAME}/${ENVIRONMENT}/$TAG[2]/client_id=$TAG[3]/year=%Y/month=%m/day=%d/%H%M%S-$UUID.gz
    s3_key_format_tag_delimiters .
    compression                 gzip
    total_file_size             1M
    upload_timeout              30s
    use_put_object              On
    auto_retry_requests         true
    net.dns.mode                TCP
[OUTPUT]
    Match                       special_logger*
    Name                        s3
    Retry_Limit                 False
    bucket                      ${RAW_LOG_BUCKET}
    s3_key_format               /${SERVICE_NAME}/${ENVIRONMENT}/$TAG/%Y%m%d%H%M%S-$UUID.gz
    s3_key_format_tag_delimiters .
    compression                 gzip
    total_file_size             1M
    upload_timeout              30s
    use_put_object              On
    auto_retry_requests         true
    net.dns.mode                TCP
[OUTPUT]
    Match                       special_logger*
    Name                        s3
    Retry_Limit                 False
    bucket                      ${ATHENA_LOG_BUCKET}
    s3_key_format               /${SERVICE_NAME}/${ENVIRONMENT}/$TAG/year=%Y/month=%m/day=%d/%H%M%S-$UUID.gz
    s3_key_format_tag_delimiters .
    compression                 gzip
    total_file_size             1M
    upload_timeout              30s
    use_put_object              On
    auto_retry_requests         true
    net.dns.mode                TCP

ssplatt on Jul 14, 2022