fluent-plugin-systemd: Corrupted pos file

Forced termination caused corruption of pos file. Mutations to pos file could be done atomically to increase reliability.

fluent-plugin-systemd version 1.0.1

From Logs

2018-07-05 07:02:41 +0000 [error]: [systemd.log] failed to read data from plugin storage file path="/var/log/systemd.log.pos" error_class=Yajl::ParseError error="lexical error: invalid char in json text.\n                                       s=97c17292555e41258bd6733e9836c\n                     (right here) ------^\n"
2018-07-05 07:02:41 +0000 [error]: config error file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError error="Unexpected error: failed to read data from plugin storage file: '/var/log/systemd.log.pos'"

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 6
Comments: 19

Most upvoted comments

Hi I am seeing a similar error reading the pos file

2018-10-18 20:38:01 +0000 [error]: failed to read data from plugin storage file path="/fluentd/pos/fluentd-journald.pos/worker0/storage.json" error_class=Fluent::ConfigError error="Invalid contents (not object) in plugin storage file: '/fluentd/pos/fluentd-journald.pos/worker0/storage.json'"
2018-10-18 20:38:01 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Unexpected error: failed to read data from plugin storage file: '/fluentd/pos/fluentd-journald.pos/worker0/storage.json'"

however in this case, the file itself exists on the host but it is empty. This is resulting in my fluentd containers crashlooping. Removing the storage.json and restarting seems to fix, will see if this issue crops up again.

nulltrope on Oct 18, 2018

Same issue here. We’ve got the file in place but it is empty. fluentd is crashing because of this.

2018-12-06 10:48:34 +0000 [error]: [in_systemd_kubelet] failed to read data from plugin storage file path="/var/log/fluentd-journald-kubelet-cursor.json" error_class=Fluent::ConfigError error="Invalid contents (not object) in plugin storage file: '/var/log/fluentd-journald-kubelet-cursor.json'"
2018-12-06 10:48:34 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Unexpected error: failed to read data from plugin storage file: '/var/log/fluentd-journald-kubelet-cursor.json'"

vladimir-kozyrev on Dec 6, 2018

2019-10-26 18:45:41 +0000 [error]: #0 failed to save data for plugin storage to file path="/var/log/fluentd-systemdlog.pos/worker0/storage.json" tmp="/var/log/fluentd-systemdlog.pos/worker0/storage.json.tmp" error_class=Errno::ENOENT error="No such file or directory @ rb_file_s_rename - (/var/log/fluentd-systemdlog.pos/worker0/storage.json.tmp, /var/log/fluentd-systemdlog.pos/worker0/storage.json)"

Getting same Error

Using below: fluent-plugin-kubernetes_metadata_filter:2.3.0
fluent-plugin-rewrite-tag-filter:2.2.0
fluent-plugin-systemd:1.0.2
fluent-plugin-vmware-loginsight:0.1.6

kumarganesh2814 on Oct 26, 2019

We are facing the same issue on centos 7 with k8s 1.12 using daemonset with fluent/fluentd-kubernetes-daemonset:v1.3.2-debian-elasticsearch: https://github.com/fluent/fluentd-kubernetes-daemonset/issues/245.

schmidtjohannes on Mar 12, 2019

I met the same issue when ‘fluentd container crashlooping’, any update?

Michake on Nov 19, 2018

we met the same issue here - ‘fluentd container crashlooping’, any update or clue to fix it?

Tim017 on Nov 13, 2018

Hi, everyone!!! We have the problem. Fluent display message

#0 [in_systemd_kubelet] failed to save data for plugin storage to file path="/var/log/fluentd-journald-kubelet-cursor.json" tmp="/var/log/fluentd-journald-kubelet-cursor.json.tmp" error_class=Errno::ENOENT error="No such file or directory @ rb_file_s_rename - (/var/log/fluentd-journald-kubelet-cursor.json.tmp, /var/log/fluentd-journald-kubelet-cursor.json)"

fluent version is fluent/fluentd-kubernetes-daemonset:v1.7-debian-elasticsearch6-1. Have you any idea??

zhasik911 on Sep 12, 2019