fluent-bit: systemd input not showing in output

I have the following configuration:

[SERVICE]                              
    Flush 1                            
    Log_Level debug                    

[INPUT]                                
    Name systemd                       
    Tag host.*                         
    Systemd_Filter _SYSTEMD_UNIT=bashdaemon.service                            

[OUTPUT]                               
    Name stdout                        
    Match *

For some reason, Fluent Bit isn’t picking up the journal entries for the bashdaemon.service unit. This screenshot below (direct link) shows this:

Top terminal: fluent-bit running the configuration
Bottom left: cat’ing out the configuration (just for verification purposes)
Bottom right: journalctl output tailing with a filter on the bashdaemon.service unit

Any thoughts on why Fluent Bit may not be ingesting these systemd unit journal entries? Or how to troubleshoot this?

Imgur

Thanks in advance!

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 29 (10 by maintainers)

Commits related to this issue

added machine-id r/o mount (see: https://github.com/fluent/fluent-bit/issues/497) — committed to sighupio/fury-kubernetes-logging by lzecca78 4 years ago
added machine-id r/o mount (see: https://github.com/fluent/fluent-bit/issues/497) — committed to sighupio/fury-kubernetes-logging by lzecca78 4 years ago

Most upvoted comments

I found a little time to dig into the source code, and the issue is obvious now.

The plugin uses systemd API (namely sd_journal_* functions), systemd itself relies on a concept of unique machine ID, stored in /etc/machine-id (see machine-id) - this is exactly the hex number in the journal directory path (I suppose this is to allow for many machines to store logs into a single shared network volume).

And now - /etc/machine-id inside a docker container is either missing or different, hence the API fails to open host journal files. A quick hack (copy /etc/machine-id from host into the container) confirmed this.

I’m not sure what is the best solution here, but mounting the file from host into container seems the best option (e.g. docker ... -v /etc/machine-id:/etc/machine-id:ro ...)

+15

codepainters on May 11, 2018

I can confirm that mounting the file /etc/machine-id read-only works. Example of what needs to be added:

      containers:
        volumeMounts:
        - name: etcmachineid
          mountPath: /etc/machine-id
          readOnly: true
      volumes:
      - name: etcmachineid
        hostPath:
          path: /etc/machine-id
          type: File

lucaim on Jun 8, 2018

can you try adding a path to the systemd journal files ?, e.g:

$ fluent-bit -i systemd -p path=/var/log/journal -p "_SYSTEMD_UNIT=sshd.service" -o stdout -f 1

edsiper on Jan 31, 2018

well, I think it depends on how each distribution set permissions in Journal files. I am happy to hear the problem has gone 😃

edsiper on Feb 7, 2018

the following command works properly:

$ fluent-bit -i systemd -p "_SYSTEMD_UNIT=sshd.service" -o stdout -f 1

note that -p needs to be right after the plugin that you want to modify the property. Adding -p after -o stdout means “set that property to stdout”.

edsiper on Jan 31, 2018