fluent-bit: [Windows] fluent-bit can not find log files grater then 2Gbytes

Bug Report

Describe the bug Using input plugin tail fluent-bit can not find log files grater then 2Gbytes

To Reproduce We have file sized 2147483648bytes = 2Gbytes

PS D:\td-agent-bit\logs> dir


    Directory: D:\td-agent-bit\logs


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----       28.05.2020     19:24     2147483648 events1.log

Starting fluent-bit

> .\fluent-bit.exe -c ..\conf\fluent-bit.conf
Fluent Bit v1.5.0
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/05/28 19:47:51] [ info] Configuration:
[2020/05/28 19:47:51] [ info]  flush time     | 5.000000 seconds
[2020/05/28 19:47:51] [ info]  grace          | 5 seconds
[2020/05/28 19:47:51] [ info]  daemon         | 0
[2020/05/28 19:47:51] [ info] ___________
[2020/05/28 19:47:51] [ info]  inputs:
[2020/05/28 19:47:51] [ info]      tail
[2020/05/28 19:47:51] [ info] ___________
[2020/05/28 19:47:51] [ info]  filters:
[2020/05/28 19:47:51] [ info] ___________
[2020/05/28 19:47:51] [ info]  outputs:
[2020/05/28 19:47:51] [ info]      es.0
[2020/05/28 19:47:51] [ info] ___________
[2020/05/28 19:47:51] [ info]  collectors:
[2020/05/28 19:47:51] [debug] [storage] [cio stream] new stream registered: tail.0
[2020/05/28 19:47:51] [ info] [storage] version=1.0.3, initializing...
[2020/05/28 19:47:51] [ info] [storage] in-memory
[2020/05/28 19:47:51] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/05/28 19:47:51] [ info] [engine] started (pid=7384)
[2020/05/28 19:47:51] [debug] [engine] coroutine stack size: 98302 bytes (96.0K)
[2020/05/28 19:47:51] [debug] [input:tail:tail.0] scanning path D:\td-agent-bit\logs\events1.log
[2020/05/28 19:47:51] [debug] [input:tail:tail.0] 0 files found for 'D:\td-agent-bit\logs\events1.log'
[2020/05/28 19:47:51] [trace] [input:tail:tail.0 at C:\projects\fluent-bit-2e87g\plugins\in_tail\tail.c:217] scan path: D:\td-agent-bit\logs\events1.log
[2020/05/28 19:47:51] [debug] [output:es:es.0] host=101.1.1.132 port=9200 uri=/_bulk index=firid-test-1 type=sid
[2020/05/28 19:47:51] [debug] [router] match rule tail.0:es.0
[2020/05/28 19:47:51] [ info] [sp] stream processor started

There is no files found.

If we reduce this file to 2147483647 bytes size output will be:

...
[2020/05/28 19:54:12] [debug] [input:tail:tail.0] 1 files found for 'D:\td-agent-bit\logs\events1.log'
[2020/05/28 19:54:12] [trace] [input:tail:tail.0 at C:\projects\fluent-bit-2e87g\plugins\in_tail\tail.c:217] scan path: D:\td-agent-bit\logs\events1.log
...

Expected behavior fluent-bit find and parse log files grater then 2Gbytes

Environment Windows Server 2016 Standart Tested on Fluent-bit 1.4.1 thue 1.5.0

[INPUT]
    Name   tail
    Tag    firid
    Parser firid
    Path   D:\td-agent-bit\logs\events*.log
    DB     ..\db\firid.db

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 15 (10 by maintainers)

Most upvoted comments

We have same issue. Someone could say when fix will be implemented?

@AlekseyKalinin I have done adding 64-bit file support to in_tail plugin.

This issue should be resolved once #2584 landed onto the main line. (I expect that the fix will be included in the next stable version: v1.6.0)