fluent-bit: Fluent Bit encountering errors after detecting log files in FSx file shares on Windows Server 2019 (v2.0.9)

This is a continuation from https://github.com/fluent/fluent-bit/issues/7185. Original file paths and other environment details/metadata has been stripped.

Problem:

After resolving the issue of Fluent Bit not detecting log files in FSx file shares on Windows Server 2019, we are now encountering errors when adding log files to bulk ingestion. It’s worth noting that the message lines with “adding log to the bulk” show the file path with the backslashes escaped out. We had to remove the backslash escaping from our INPUT stanza to get Fluent Bit to detect the log files in the first place.

Environment:

• Fluent Bit Version: 2.0.9
• Operating System: Windows Server 2019
• FSx File Share Path: \\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs

Steps to Reproduce:

• Configure Fluent Bit with the following input stanza:

   [INPUT]
   Name tail
   Path \\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\*.txt
   Path_Key On
   Tag app_logs
   Buffer_Max_Size 1024k
   Read_from_Head On

• Start Fluent Bit with trace log level.

• Create or modify log files in the specified FSx file share path.

• Observe that Fluent Bit does detect log files but begins throwing errors:

2023/04/18 17:07:36] [ info] [sp] stream processor started
[2023/04/18 17:07:36] [debug] [input:tail:tail.0] inode=281474976734757 file=\\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\example.txt promote to TAIL_EVENT
[2023/04/18 17:07:36] [ info] [output:logzio:logzio.0] worker #0 started
[2023/04/18 17:07:36] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:36] [debug] [input:tail:tail.0] inode=3659174697240457 file=\\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\example.txt.1 promote to TAIL_EVENT
[2023/04/18 17:07:36] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:36] [debug] [input:tail:tail.0] inode=6755399441059732 file=\\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\example.txt.2 promote to TAIL_EVENT
[2023/04/18 17:07:36] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:36] [debug] [input:tail:tail.0] inode=281474976734381 file=\\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\example.txt.3 promote to TAIL_EVENT
[2023/04/18 17:07:36] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:36] [debug] [input:tail:tail.0] inode=281474976734386 file=\\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\example.txt.4 promote to TAIL_EVENT
[2023/04/18 17:07:36] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:36] [debug] [input:tail:tail.0] inode=281474976734391 file=\\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\example.txt.5 promote to TAIL_EVENT
[2023/04/18 17:07:36] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
2023/04/18 17:07:37 [logzio] adding log to the bulk: {"@timestamp":"2023-04-18T17:07:36.7501512-06:00","On":"\\\\fsx-example-host.example.net\\share\\staging\\if\\dir\\example-server\\Logs\\example.txt","account_id":"","ami_id":"","az":"us-east-1b","ec2_instance_id":"","ec2_instance_type":"","fluentbit_tag":"app_logs","host":"","hostname":"","log":"2023-04-18 17:06:54,191 [18] INFO  IFLog [(null)] - Server Info request from ","output_id":"logzio_output_1","private_ip":"","type":"logzio-fluent-bit","vpc_id":""}
2023/04/18 17:07:37 [logzio] adding log to the bulk: {"@timestamp":"2023-04-18T17:07:36.7501547-06:00","On":"\\\\fsx-example-host.example.net\\share\\staging\\if\\dir\\example-server\\Logs\\example.txt","account_id":"","ami_id":"","az":"us-east-1b","ec2_instance_id":"","ec2_instance_type":"","fluentbit_tag":"app_logs","host":"","hostname":"","log":"2023-04-18 17:06:54,644 [5] INFO  IFLog [(null)] - Server Info request from ","output_id":"logzio_output_1","private_ip":"","type":"logzio-fluent-bit","vpc_id":""}
2023/04/18 17:07:37 [logzio] adding log to the bulk: {"@timestamp":"2023-04-18T17:07:36.7501551-06:00","On":"\\\\fsx-example-host.example.net\\share\\staging\\if\\dir\\example-server\\Logs\\example.txt","account_id":"","ami_id":"","az":"us-east-1b","ec2_instance_id":"","ec2_instance_type":"","fluentbit_tag":"app_logs","host":"","hostname":"","log":"2023-04-18 17:07:24,226 [11] INFO  IFLog [(null)] - Server Info request from ","output_id":"logzio_output_1","private_ip":"","type":"logzio-fluent-bit","vpc_id":""}
2023/04/18 17:07:37 [logzio] adding log to the bulk: {"@timestamp":"2023-04-18T17:07:36.7501555-06:00","On":"\\\\fsx-example-host.example.net\\share\\staging\\if\\dir\\example-server\\Logs\\example.txt","account_id":"","ami_id":"","az":"us-east-1b","ec2_instance_id":"","vpc_id":""}
2023/04/18 17:07:38 [logzio] successfully sent bulk to logz.io
[2023/04/18 17:07:38] [debug] [out flush] cb_destroy coro_id=0
[2023/04/18 17:07:38] [debug] [task] destroy task=000001BBFAB2D750 (task_id=0)
[2023/04/18 17:07:38] [debug] [input chunk] update output instances with new chunk size diff=448
[2023/04/18 17:07:39] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:39] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:39] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:39] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:39] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:39] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory

Expected Behavior
Fluent Bit should detect and send logs to output without errors.

Actual Behavior
Fluent Bit does detect file changes in FSx file shares on Windows Server 2019 but throws many errors.

Additional Context
We have validated that the FSx file share path is accessible from the Windows Server 2019 system, and we are able to read the content of the files without any issues.

Please let me know if you need any further information or if there are any workarounds or solutions to this issue.