fluentd: Fluentd crashes

Running fluentd 0.14.1, installed with gem, on Arch Linux. I setup a simple fluent.conf demo like this:

<source>
  @type forward
  port 24224
</source>

<filter demo.temperature>
  @type record_transformer
  enable_ruby true
  <record>
    cpu_temp ${record["cpu_temp"] + 273.1}
  </record>
</filter>

<match demo.temperature>
  @type file
  path temperature
  flush_interval 1s
  append true
</match>

<match **>
  @type stdout
</match>

And I fed it data with a python script as follows:

#!/usr/bin/env python
import threading
from random import random
from fluent import event, sender


def set_interval(func, sec):
    def func_wrapper():
        set_interval(func, sec)
        func()
    t = threading.Timer(sec, func_wrapper)
    t.start()
    return t


def send_event():
    temp = 70 + random() * (120 - 70)  # 70 to 100
    e = {"hostname": "somehost", "cpu_temp": temp}
    event.Event("temperature", e)

    print(e)


if __name__ == "__main__":
    sender.setup('demo', host='localhost', port=24224)
    set_interval(send_event, 1)

However, fluentd goes crazy after a short while. The directory with the fluentd configuration file gets filled with buffer files:

demo.py
fluentd.conf
requirements.txt
temperature.20160713.log
temperature.b5377a1127d7372b0cce743309564537e.log
temperature.b5377a1127d7372b0cce743309564537e.log.meta
temperature.q5377a0ef3166873b918670336db4363b.log
temperature.q5377a0ef3166873b918670336db4363b.log.meta
temperature.q5377a0f0259024f31fbd332a6b46d5d0.log
temperature.q5377a0f0259024f31fbd332a6b46d5d0.log.meta
temperature.q5377a0f119b896c8c47a246b8c5beb3e.log
temperature.q5377a0f119b896c8c47a246b8c5beb3e.log.meta
temperature.q5377a0f20df0d19d406ccd7bfce1b7ac.log
temperature.q5377a0f20df0d19d406ccd7bfce1b7ac.log.meta
temperature.q5377a0f3023b750a28abbe69ad70814a.log
temperature.q5377a0f3023b750a28abbe69ad70814a.log.meta
temperature.q5377a0f3f65fa1859ba3e9d757e6c354.log
temperature.q5377a0f3f65fa1859ba3e9d757e6c354.log.meta
temperature.q5377a0f4ea975c703d1a09a1851b4941.log
temperature.q5377a0f4ea975c703d1a09a1851b4941.log.meta
temperature.q5377a0f5ded4e986152edef1f0ac5a25.log
temperature.q5377a0f5ded4e986152edef1f0ac5a25.log.meta
temperature.q5377a0f6d30d0540bc30e18159e6ba58.log
temperature.q5377a0f6d30d0540bc30e18159e6ba58.log.meta
temperature.q5377a0f7c7429d5f36b185742e212109.log
temperature.q5377a0f7c7429d5f36b185742e212109.log.meta
temperature.q5377a0f8bb79791d30682c34af3c1427.log
temperature.q5377a0f8bb79791d30682c34af3c1427.log.meta
temperature.q5377a0f9afbe277a9716c4c9d0665545.log
temperature.q5377a0f9afbe277a9716c4c9d0665545.log.meta
temperature.q5377a0faa422051a5e377990842dc252.log
temperature.q5377a0faa422051a5e377990842dc252.log.meta
temperature.q5377a0fb983431366edd54cb8ce22488.log
temperature.q5377a0fb983431366edd54cb8ce22488.log.meta
temperature.q5377a0fc8c6b774ffe008bf0c5f57037.log
temperature.q5377a0fc8c6b774ffe008bf0c5f57037.log.meta
temperature.q5377a0fd80a63a9cf2747d50ee657932.log
temperature.q5377a0fd80a63a9cf2747d50ee657932.log.meta
temperature.q5377a0fe74dc694dd158ad01cd33660e.log
temperature.q5377a0fe74dc694dd158ad01cd33660e.log.meta
temperature.q5377a0ff691f91a3e90fc0249bc90f73.log
temperature.q5377a0ff691f91a3e90fc0249bc90f73.log.meta
temperature.q5377a1005d45b5e1839042e134b95a4b.log
temperature.q5377a1005d45b5e1839042e134b95a4b.log.meta
temperature.q5377a101518dba28a4d1fa385cb81e67.log
# and many more like this

Trying to close fluentd with SIGINT or SIGTERM fails, so I killed it with SIGKILL:

2016-07-13 10:10:59 +0900 [info]: plugin/in_forward.rb:99:listen: listening fluent socket on 0.0.0.0:24224
2016-07-13 10:10:59 +0900 [trace]: plugin/in_forward.rb:246:initialize: accepted fluent socket from '127.0.0.1:49676': object_id=12974500
2016-07-13 10:10:59 +0900 fluent.trace: {"message":"accepted fluent socket from '127.0.0.1:49676': object_id=12974500"}
^C2016-07-13 10:11:36 +0900 [debug]: fluent/supervisor.rb:489:block in install_main_process_signal_handlers: fluentd main process get SIGINT
2016-07-13 10:11:36 +0900 [info]: serverengine/server.rb:43:stop: Received graceful stop
2016-07-13 10:11:36 +0900 fluent.debug: {"message":"fluentd main process get SIGINT"}
2016-07-13 10:11:37 +0900 [debug]: fluent/supervisor.rb:493:block in install_main_process_signal_handlers: fluentd main process get SIGTERM
2016-07-13 10:11:37 +0900 [debug]: fluent/supervisor.rb:496:block in install_main_process_signal_handlers: getting start to shutdown main process
2016-07-13 10:11:37 +0900 [info]: fluent/engine.rb:190:run: shutting down fluentd
2016-07-13 10:11:37 +0900 [debug]: fluent/root_agent.rb:170:block (2 levels) in shutdown: calling stop on input plugin type=:forward plugin_id="object:cb5700"
2016-07-13 10:11:37 +0900 [debug]: fluent/root_agent.rb:170:block (2 levels) in shutdown: calling stop on output plugin type=:file plugin_id="object:a9b154"
2016-07-13 10:11:37 +0900 [debug]: fluent/root_agent.rb:170:block (2 levels) in shutdown: calling stop on filter plugin type=:record_transformer plugin_id="object:edecac"
2016-07-13 10:11:37 +0900 [debug]: fluent/root_agent.rb:170:block (2 levels) in shutdown: calling stop on output plugin type=:stdout plugin_id="object:f683e4"
2016-07-13 10:11:37 +0900 fluent.debug: {"message":"fluentd main process get SIGTERM"}
2016-07-13 10:11:37 +0900 fluent.debug: {"message":"getting start to shutdown main process"}
2016-07-13 10:11:37 +0900 fluent.info: {"message":"shutting down fluentd"}
2016-07-13 10:11:37 +0900 fluent.debug: {"type":"forward","plugin_id":"object:cb5700","message":"calling stop on input plugin type=:forward plugin_id=\"object:cb5700\""}
2016-07-13 10:11:37 +0900 fluent.debug: {"type":"file","plugin_id":"object:a9b154","message":"calling stop on output plugin type=:file plugin_id=\"object:a9b154\""}
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: preparing shutdown input plugin type=:forward plugin_id="object:cb5700"
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: preparing shutdown output plugin type=:file plugin_id="object:a9b154"
2016-07-13 10:11:37 +0900 fluent.debug: {"type":"record_transformer","plugin_id":"object:edecac","message":"calling stop on filter plugin type=:record_transformer plugin_id=\"object:edecac\""}
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: preparing shutdown filter plugin type=:record_transformer plugin_id="object:edecac"
2016-07-13 10:11:37 +0900 fluent.debug: {"type":"stdout","plugin_id":"object:f683e4","message":"calling stop on output plugin type=:stdout plugin_id=\"object:f683e4\""}
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: preparing shutdown output plugin type=:stdout plugin_id="object:f683e4"
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: shutting down input plugin type=:forward plugin_id="object:cb5700"
2016-07-13 10:11:37 +0900 fluent.info: {"type":"forward","plugin_id":"object:cb5700","message":"preparing shutdown input plugin type=:forward plugin_id=\"object:cb5700\""}
2016-07-13 10:11:37 +0900 fluent.info: {"type":"file","plugin_id":"object:a9b154","message":"preparing shutdown output plugin type=:file plugin_id=\"object:a9b154\""}
2016-07-13 10:11:37 +0900 fluent.info: {"type":"record_transformer","plugin_id":"object:edecac","message":"preparing shutdown filter plugin type=:record_transformer plugin_id=\"object:edecac\""}
2016-07-13 10:11:37 +0900 fluent.info: {"type":"stdout","plugin_id":"object:f683e4","message":"preparing shutdown output plugin type=:stdout plugin_id=\"object:f683e4\""}
2016-07-13 10:11:37 +0900 fluent.info: {"type":"forward","plugin_id":"object:cb5700","message":"shutting down input plugin type=:forward plugin_id=\"object:cb5700\""}
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: shutting down output plugin type=:file plugin_id="object:a9b154"
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: shutting down filter plugin type=:record_transformer plugin_id="object:edecac"
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: shutting down output plugin type=:stdout plugin_id="object:f683e4"
2016-07-13 10:11:37 +0900 [debug]: fluent/root_agent.rb:170:block (2 levels) in shutdown: calling after_shutdown on input plugin type=:forward plugin_id="object:cb5700"
2016-07-13 10:11:37 +0900 [debug]: fluent/root_agent.rb:170:block (2 levels) in shutdown: calling after_shutdown on output plugin type=:file plugin_id="object:a9b154"
2016-07-13 10:11:37 +0900 fluent.info: {"type":"file","plugin_id":"object:a9b154","message":"shutting down output plugin type=:file plugin_id=\"object:a9b154\""}
2016-07-13 10:11:37 +0900 fluent.info: {"type":"record_transformer","plugin_id":"object:edecac","message":"shutting down filter plugin type=:record_transformer plugin_id=\"object:edecac\""}
2016-07-13 10:11:37 +0900 fluent.info: {"type":"stdout","plugin_id":"object:f683e4","message":"shutting down output plugin type=:stdout plugin_id=\"object:f683e4\""}
2016-07-13 10:11:37 +0900 fluent.debug: {"type":"forward","plugin_id":"object:cb5700","message":"calling after_shutdown on input plugin type=:forward plugin_id=\"object:cb5700\""}
2016-07-13 10:11:37 +0900 fluent.debug: {"type":"file","plugin_id":"object:a9b154","message":"calling after_shutdown on output plugin type=:file plugin_id=\"object:a9b154\""}
[1]    26900 killed     fluentd -c ./fluentd.conf -vv

If I let it keep running it crashes my computer. Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 18 (9 by maintainers)

Most upvoted comments

I’ll fix this problem at next version (v0.14.2). The workaround for this issue is to use different directly between path and buffer path.