fluent-bit: FluentBit crash on Windows

Bug Report

Describe the bug

Following up from https://github.com/fluent/fluent-bit/commit/56d619fa73f2be9f0b7a5ae040d86ef0b9734a1c#commitcomment-39848013, I tried running the current latest fluent-bit, version 1.4.6, but it crashed a few times several minutes after starting (as a Windows service with winsw).

Your Environment

Version used: 1.4.6
Configuration: See below
Environment name and version: AWS EC2 instance
Server type and version: Windows
Operating System and version: Server 2016
Filters and plugins: in_tail, filter_lua, output_es

fluent-bit.conf:

[SERVICE]
    daemon                  false
    flush                   1
    log_level               debug
    parsers_file            parsers.conf
    plugins_file            plugins.conf
    http_server             true
    http_listen             0.0.0.0
    http_port               2020

[INPUT]
    name                    tail
    tag                     pharos.*
    path                    C:\PharosSystems\Logs\Json\PharosMpsLog.txt
    db                      C:\PharosSystems\Logs\Json\FluentBit.db
    parser                  json
    buffer_chunk_size       64k
    buffer_max_size         64k
    mem_buf_limit           8m
    skip_long_lines         true
    refresh_interval        10

[FILTER]
    name                    lua
    match                   *
    script                  filters.lua
    call                    transform

[OUTPUT]
    name                    es
    match                   *
    host                    logs
    port                    80
    logstash_format         true
    type                    _doc
    replace_dots            true
    retry_limit             5
    trace_error             true

filters.lua:

function transform(tag, timestamp, record)
    if record["Payload"] then
        record["Service"] = record["Payload"]["processName"]
    end
	
    if not record["Message"] and record["Payload"]["message"] then
        record["Message"], record["Payload"]["message"] = record["Payload"]["message"], nil
    end
    return 1, timestamp, record
end

Dump file captured with Sysinternals ProcDump using procdump64.exe -ma -e fluent-bit.exe:

fluent-bit.exe_200612_053456.zip

Note: missing symbols for fluent-bit.exe; @fujimotos could we please include PDB symbols with future Windows build artifacts? 🙏

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 23 (23 by maintainers)

Most upvoted comments

@gitfool Great! Now I’ll prepare to push these fixes to mainline. So thankful for your investigation and testing!

fujimotos on Jun 19, 2020

… and I just notice you already did get the dump in the https://github.com/fluent/fluent-bit/issues/2251#issuecomment-644327972.

I’ll take a closer look at your log tomorrow and investigate the root cause of the crash issue.

fujimotos on Jun 16, 2020

@gitfool I preapred a branch win32-2251 for your testing. It’s basically a set of small Windows patches on top of v1.4.6.

Commit: https://github.com/fluent/fluent-bit/commits/win32-2251
AppVeyor Build: td-agent-bit-1.4.6-win64 (AppVeyor)

The testing package is available from the second link; It includes a correct PDB file, so you can readily test things out.

I’m not sure but I think the symbols file is for fluent-bit.dll (not fluent-bit.exe), which I why I had trouble loading it, and also why I suspect the stack frames look wonky.

That’s a good catch. I took it account into f1810e2d and made it sure that both EXE and DLL have separate PDB files.

The branch win32-2251 includes this fix, and it now produces the following file assets:

build/bin/
├── fluent-bit.dll
├── fluent-bit.dll.pdb
├── fluent-bit.exe
└── fluent-bit.pdb

Please feel free to tell me if you notice anything.

fujimotos on Jun 16, 2020