fluentd: Fluentd crashes
Running fluentd 0.14.1, installed with gem
, on Arch Linux. I setup a simple fluent.conf
demo like this:
<source>
@type forward
port 24224
</source>
<filter demo.temperature>
@type record_transformer
enable_ruby true
<record>
cpu_temp ${record["cpu_temp"] + 273.1}
</record>
</filter>
<match demo.temperature>
@type file
path temperature
flush_interval 1s
append true
</match>
<match **>
@type stdout
</match>
And I fed it data with a python script as follows:
#!/usr/bin/env python
import threading
from random import random
from fluent import event, sender
def set_interval(func, sec):
def func_wrapper():
set_interval(func, sec)
func()
t = threading.Timer(sec, func_wrapper)
t.start()
return t
def send_event():
temp = 70 + random() * (120 - 70) # 70 to 100
e = {"hostname": "somehost", "cpu_temp": temp}
event.Event("temperature", e)
print(e)
if __name__ == "__main__":
sender.setup('demo', host='localhost', port=24224)
set_interval(send_event, 1)
However, fluentd goes crazy after a short while. The directory with the fluentd configuration file gets filled with buffer files:
demo.py
fluentd.conf
requirements.txt
temperature.20160713.log
temperature.b5377a1127d7372b0cce743309564537e.log
temperature.b5377a1127d7372b0cce743309564537e.log.meta
temperature.q5377a0ef3166873b918670336db4363b.log
temperature.q5377a0ef3166873b918670336db4363b.log.meta
temperature.q5377a0f0259024f31fbd332a6b46d5d0.log
temperature.q5377a0f0259024f31fbd332a6b46d5d0.log.meta
temperature.q5377a0f119b896c8c47a246b8c5beb3e.log
temperature.q5377a0f119b896c8c47a246b8c5beb3e.log.meta
temperature.q5377a0f20df0d19d406ccd7bfce1b7ac.log
temperature.q5377a0f20df0d19d406ccd7bfce1b7ac.log.meta
temperature.q5377a0f3023b750a28abbe69ad70814a.log
temperature.q5377a0f3023b750a28abbe69ad70814a.log.meta
temperature.q5377a0f3f65fa1859ba3e9d757e6c354.log
temperature.q5377a0f3f65fa1859ba3e9d757e6c354.log.meta
temperature.q5377a0f4ea975c703d1a09a1851b4941.log
temperature.q5377a0f4ea975c703d1a09a1851b4941.log.meta
temperature.q5377a0f5ded4e986152edef1f0ac5a25.log
temperature.q5377a0f5ded4e986152edef1f0ac5a25.log.meta
temperature.q5377a0f6d30d0540bc30e18159e6ba58.log
temperature.q5377a0f6d30d0540bc30e18159e6ba58.log.meta
temperature.q5377a0f7c7429d5f36b185742e212109.log
temperature.q5377a0f7c7429d5f36b185742e212109.log.meta
temperature.q5377a0f8bb79791d30682c34af3c1427.log
temperature.q5377a0f8bb79791d30682c34af3c1427.log.meta
temperature.q5377a0f9afbe277a9716c4c9d0665545.log
temperature.q5377a0f9afbe277a9716c4c9d0665545.log.meta
temperature.q5377a0faa422051a5e377990842dc252.log
temperature.q5377a0faa422051a5e377990842dc252.log.meta
temperature.q5377a0fb983431366edd54cb8ce22488.log
temperature.q5377a0fb983431366edd54cb8ce22488.log.meta
temperature.q5377a0fc8c6b774ffe008bf0c5f57037.log
temperature.q5377a0fc8c6b774ffe008bf0c5f57037.log.meta
temperature.q5377a0fd80a63a9cf2747d50ee657932.log
temperature.q5377a0fd80a63a9cf2747d50ee657932.log.meta
temperature.q5377a0fe74dc694dd158ad01cd33660e.log
temperature.q5377a0fe74dc694dd158ad01cd33660e.log.meta
temperature.q5377a0ff691f91a3e90fc0249bc90f73.log
temperature.q5377a0ff691f91a3e90fc0249bc90f73.log.meta
temperature.q5377a1005d45b5e1839042e134b95a4b.log
temperature.q5377a1005d45b5e1839042e134b95a4b.log.meta
temperature.q5377a101518dba28a4d1fa385cb81e67.log
# and many more like this
Trying to close fluentd with SIGINT or SIGTERM fails, so I killed it with SIGKILL:
2016-07-13 10:10:59 +0900 [info]: plugin/in_forward.rb:99:listen: listening fluent socket on 0.0.0.0:24224
2016-07-13 10:10:59 +0900 [trace]: plugin/in_forward.rb:246:initialize: accepted fluent socket from '127.0.0.1:49676': object_id=12974500
2016-07-13 10:10:59 +0900 fluent.trace: {"message":"accepted fluent socket from '127.0.0.1:49676': object_id=12974500"}
^C2016-07-13 10:11:36 +0900 [debug]: fluent/supervisor.rb:489:block in install_main_process_signal_handlers: fluentd main process get SIGINT
2016-07-13 10:11:36 +0900 [info]: serverengine/server.rb:43:stop: Received graceful stop
2016-07-13 10:11:36 +0900 fluent.debug: {"message":"fluentd main process get SIGINT"}
2016-07-13 10:11:37 +0900 [debug]: fluent/supervisor.rb:493:block in install_main_process_signal_handlers: fluentd main process get SIGTERM
2016-07-13 10:11:37 +0900 [debug]: fluent/supervisor.rb:496:block in install_main_process_signal_handlers: getting start to shutdown main process
2016-07-13 10:11:37 +0900 [info]: fluent/engine.rb:190:run: shutting down fluentd
2016-07-13 10:11:37 +0900 [debug]: fluent/root_agent.rb:170:block (2 levels) in shutdown: calling stop on input plugin type=:forward plugin_id="object:cb5700"
2016-07-13 10:11:37 +0900 [debug]: fluent/root_agent.rb:170:block (2 levels) in shutdown: calling stop on output plugin type=:file plugin_id="object:a9b154"
2016-07-13 10:11:37 +0900 [debug]: fluent/root_agent.rb:170:block (2 levels) in shutdown: calling stop on filter plugin type=:record_transformer plugin_id="object:edecac"
2016-07-13 10:11:37 +0900 [debug]: fluent/root_agent.rb:170:block (2 levels) in shutdown: calling stop on output plugin type=:stdout plugin_id="object:f683e4"
2016-07-13 10:11:37 +0900 fluent.debug: {"message":"fluentd main process get SIGTERM"}
2016-07-13 10:11:37 +0900 fluent.debug: {"message":"getting start to shutdown main process"}
2016-07-13 10:11:37 +0900 fluent.info: {"message":"shutting down fluentd"}
2016-07-13 10:11:37 +0900 fluent.debug: {"type":"forward","plugin_id":"object:cb5700","message":"calling stop on input plugin type=:forward plugin_id=\"object:cb5700\""}
2016-07-13 10:11:37 +0900 fluent.debug: {"type":"file","plugin_id":"object:a9b154","message":"calling stop on output plugin type=:file plugin_id=\"object:a9b154\""}
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: preparing shutdown input plugin type=:forward plugin_id="object:cb5700"
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: preparing shutdown output plugin type=:file plugin_id="object:a9b154"
2016-07-13 10:11:37 +0900 fluent.debug: {"type":"record_transformer","plugin_id":"object:edecac","message":"calling stop on filter plugin type=:record_transformer plugin_id=\"object:edecac\""}
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: preparing shutdown filter plugin type=:record_transformer plugin_id="object:edecac"
2016-07-13 10:11:37 +0900 fluent.debug: {"type":"stdout","plugin_id":"object:f683e4","message":"calling stop on output plugin type=:stdout plugin_id=\"object:f683e4\""}
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: preparing shutdown output plugin type=:stdout plugin_id="object:f683e4"
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: shutting down input plugin type=:forward plugin_id="object:cb5700"
2016-07-13 10:11:37 +0900 fluent.info: {"type":"forward","plugin_id":"object:cb5700","message":"preparing shutdown input plugin type=:forward plugin_id=\"object:cb5700\""}
2016-07-13 10:11:37 +0900 fluent.info: {"type":"file","plugin_id":"object:a9b154","message":"preparing shutdown output plugin type=:file plugin_id=\"object:a9b154\""}
2016-07-13 10:11:37 +0900 fluent.info: {"type":"record_transformer","plugin_id":"object:edecac","message":"preparing shutdown filter plugin type=:record_transformer plugin_id=\"object:edecac\""}
2016-07-13 10:11:37 +0900 fluent.info: {"type":"stdout","plugin_id":"object:f683e4","message":"preparing shutdown output plugin type=:stdout plugin_id=\"object:f683e4\""}
2016-07-13 10:11:37 +0900 fluent.info: {"type":"forward","plugin_id":"object:cb5700","message":"shutting down input plugin type=:forward plugin_id=\"object:cb5700\""}
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: shutting down output plugin type=:file plugin_id="object:a9b154"
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: shutting down filter plugin type=:record_transformer plugin_id="object:edecac"
2016-07-13 10:11:37 +0900 [info]: fluent/root_agent.rb:196:block (3 levels) in shutdown: shutting down output plugin type=:stdout plugin_id="object:f683e4"
2016-07-13 10:11:37 +0900 [debug]: fluent/root_agent.rb:170:block (2 levels) in shutdown: calling after_shutdown on input plugin type=:forward plugin_id="object:cb5700"
2016-07-13 10:11:37 +0900 [debug]: fluent/root_agent.rb:170:block (2 levels) in shutdown: calling after_shutdown on output plugin type=:file plugin_id="object:a9b154"
2016-07-13 10:11:37 +0900 fluent.info: {"type":"file","plugin_id":"object:a9b154","message":"shutting down output plugin type=:file plugin_id=\"object:a9b154\""}
2016-07-13 10:11:37 +0900 fluent.info: {"type":"record_transformer","plugin_id":"object:edecac","message":"shutting down filter plugin type=:record_transformer plugin_id=\"object:edecac\""}
2016-07-13 10:11:37 +0900 fluent.info: {"type":"stdout","plugin_id":"object:f683e4","message":"shutting down output plugin type=:stdout plugin_id=\"object:f683e4\""}
2016-07-13 10:11:37 +0900 fluent.debug: {"type":"forward","plugin_id":"object:cb5700","message":"calling after_shutdown on input plugin type=:forward plugin_id=\"object:cb5700\""}
2016-07-13 10:11:37 +0900 fluent.debug: {"type":"file","plugin_id":"object:a9b154","message":"calling after_shutdown on output plugin type=:file plugin_id=\"object:a9b154\""}
[1] 26900 killed fluentd -c ./fluentd.conf -vv
If I let it keep running it crashes my computer. Thanks!
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 18 (9 by maintainers)
I’ll fix this problem at next version (v0.14.2). The workaround for this issue is to use different directly between path and buffer path.