fluent-bit: systemd input plugin cannot read zstd compressed/hash collision hardened journal files in systemd >= 246

Bug Report

Describe the bug

systemd 246 changed the default format of journal files:

    * systemd-journald gained support for zstd compression of large fields
      in journal files. The hash tables in journal files have been hardened
      against hash collisions. This is an incompatible change and means
      that journal files created with new systemd versions are not readable
      with old versions. If the $SYSTEMD_JOURNAL_KEYED_HASH boolean
      environment variable for systemd-journald.service is set to 0 this
      new hardening functionality may be turned off, so that generated
      journal files remain compatible with older journalctl
      implementations.

See https://github.com/systemd/systemd/blob/v246/NEWS#L323-L331 and https://www.freedesktop.org/software/systemd/man/journald.conf.html.

The systemd input plugin is unable to read these files unless SYSTEMD_JOURNAL_KEYED_HASH=0 (to disable the hash table hardening) and Compress=false (to disable compression of large fields).

To Reproduce

Steps to reproduce the problem:

Configure a systemd input plugin in fluent bit
Run fluent bit on a system using systemd >= 246 without SYSTEMD_JOURNAL_KEYED_HASH=0 and Compress=false
Observe that no records are emitted to output plugins and that metrics show no records/bytes are being ingested by the input plugin.

Expected behavior

The systemd input plugin should be able to parse any valid journal file format. Alternatively, the behavior and workaround should be documented in the input plugin’s documentation.

Screenshots

Your Environment

Version used: 1.5.4
Configuration:
Environment name and version (e.g. Kubernetes? What version?):
Server type and version:
Operating System and version: Flatcar Linux 2605.9.0+
Filters and plugins: systemd input

Additional context

This was tricky to spot since Fluent Bit did not log any errors attempting to read the new files. The best way to detect this is to examine the input plugin metrics.

We’re running an older version of Fluent Bit but the systemd input plugin hasn’t changed significantly since that version.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 19
Comments: 21 (11 by maintainers)

Most upvoted comments

Hello folks,

reaching out to you from the Flatcar Container Linux maintainers team to get an update on this issue. The issue impacts a number of our users; we maintain a tracking issue in our project.

We understand keyed hashes are a security / integrity feature in journald to mitigate a hash collision vulnerability of earlier versions. Currently, the only workaround available to Flatcar users to keep fluentd working is to switch off this feature, exposing their journals to potential manipulation, and making the journal less usable for e.g. auditing purposes.

I noticed there’s a PR open to fix the root cause, updating the systemd libraries in fluentd’s docker image to a version which is capable of parsing the new journal. Also, this issue appears to be on the roadmap for the fluent bit 1.7 release.

Is there anything we can help you with to get more traction with ingesting the fix into fluent bit?

+13

t-lo on Jun 29, 2021

Remove kale

sedlund on Aug 31, 2021

FWIW, our solution was to customize the docker file for 1.6.x, switching the builder from buster-slim to buster-backports, so that the container had systemd v247.x shared libraries inside of it instead of v241.x shared libraries.

chadcatlett on Feb 12, 2021

I’ve updated PR #3177.

As pointed out by @t-lo, in its current state using docker images provided by fluentbit is problematic on systems with newer versions of systemd. This issue and PR have been open for some time. It would be really appreciated in getting some feedback.

Even if my PR isn’t accepted, I’d just be happy knowing that something is in the pipeline to fix this issue.

yasn77 on Sep 8, 2021