fluentd: Why is path to pos_file required when the path to the log file itself does not exist?

If the path to the log file does not exist, fluentd will happily wait for the path to get created and will start watching the file:

# /usr/bin/fluentd -c example.conf 
2016-04-01 20:29:47 +0000 [info]: reading config file path="example.conf"
2016-04-01 20:29:47 +0000 [info]: starting fluentd-0.12.5
2016-04-01 20:29:47 +0000 [info]: gem 'fluent-plugin-add' version '0.0.3'
2016-04-01 20:29:47 +0000 [info]: gem 'fluentd' version '0.12.5'
2016-04-01 20:29:47 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path /var/log/_nonexist_/my.log
    tag foobar
    format /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}) (?<message>.*)/
    time_format %F %T.%L
  </source>
  <match **>
    type stdout
    output_type json
  </match>
</ROOT>
2016-04-01 20:29:47 +0000 [info]: adding match pattern="**" type="stdout"
2016-04-01 20:29:47 +0000 [info]: adding source type="tail"
2016-04-01 20:29:47 +0000 [warn]: 'pos_file PATH' parameter is not set to a 'tail' source.
2016-04-01 20:29:47 +0000 [warn]: this parameter is highly recommended to save the position to resume tailing.
2016-04-02 20:30:27 +0000 [info]: following tail of /var/log/_nonexist_/my.log
2016-04-02 20:30:27 +0000 fluent.info: {"message":"following tail of /var/log/_nonexist_/my.log"}

However, if the configuration specifies pos_file and the directory does not exist, fluentd errors out with ENOENT:

# /usr/bin/fluentd -c example.conf 
2016-04-01 20:33:30 +0000 [info]: reading config file path="example.conf"
2016-04-01 20:33:30 +0000 [info]: starting fluentd-0.12.5
2016-04-01 20:33:30 +0000 [info]: gem 'fluent-plugin-add' version '0.0.3'
2016-04-01 20:33:30 +0000 [info]: gem 'fluentd' version '0.12.5'
2016-04-01 20:33:30 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path /var/log/_nonexist_/my.log
    pos_file /var/log/_nonexist_/my.log.pos
    tag foobar
    format /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}) (?<message>.*)/
    time_format %F %T.%L
  </source>
  <match **>
    type stdout
    output_type json
  </match>
</ROOT>
2016-04-01 20:33:30 +0000 [info]: adding match pattern="**" type="stdout"
2016-04-01 20:33:30 +0000 [info]: adding source type="tail"
2016-04-01 20:33:30 +0000 [error]: unexpected error error_class=Errno::ENOENT error=#<Errno::ENOENT: No such file or directory - /var/log/_nonexist_/my.log.pos>
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/plugin/in_tail.rb:77:in `initialize'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/plugin/in_tail.rb:77:in `open'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/plugin/in_tail.rb:77:in `start'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/root_agent.rb:111:in `block in start'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/root_agent.rb:110:in `each'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/root_agent.rb:110:in `start'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/engine.rb:201:in `start'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/engine.rb:151:in `run'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/supervisor.rb:481:in `run_engine'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/supervisor.rb:140:in `block in start'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/supervisor.rb:266:in `call'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/supervisor.rb:266:in `main_process'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/supervisor.rb:241:in `block in supervise'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/supervisor.rb:240:in `fork'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/supervisor.rb:240:in `supervise'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/supervisor.rb:134:in `start'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/lib/fluent/command/fluentd.rb:167:in `<top (required)>'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-04-01 20:33:30 +0000 [error]: /usr/share/gems/gems/fluentd-0.12.5/bin/fluentd:6:in `<top (required)>'
  2016-04-01 20:33:30 +0000 [error]: /usr/bin/fluentd:23:in `load'
  2016-04-01 20:33:30 +0000 [error]: /usr/bin/fluentd:23:in `<main>'
2016-04-01 20:33:30 +0000 [info]: shutting down fluentd
2016-04-01 20:33:30 +0000 [info]: process finished code=0
2016-04-01 20:33:30 +0000 [warn]: process died within 1 second. exit.

If fluentd knows when the file comes into existence, why can’t it wait until the point to create the pos_file?

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 20 (11 by maintainers)

Commits related to this issue

Ensure pos log directory exists before starting fluentd. See fluent/fluentd#879 — committed to broadinstitute/clio by danxmoran 7 years ago
Ensure pos log directory exists before starting fluentd. See fluent/fluentd#879 — committed to broadinstitute/clio by danxmoran 7 years ago
Ensure pos log directory exists before starting fluentd. See fluent/fluentd#879 — committed to broadinstitute/clio by danxmoran 7 years ago
Ensure pos log directory exists before starting fluentd. See fluent/fluentd#879 — committed to broadinstitute/clio by danxmoran 7 years ago
DSDEGP-1419 Modify deploy logic to support deploying to staging (#54) * Ensure pos log directory exists before starting fluentd (See fluent/fluentd#879). * Move Clio base version into file outside o... — committed to broadinstitute/clio by danxmoran 7 years ago
DSDEGP-1419 Modify deploy logic to support deploying to staging (#54) * Ensure pos log directory exists before starting fluentd (See fluent/fluentd#879). * Move Clio base version into file outside o... — committed to broadinstitute/clio by danxmoran 7 years ago
DSDEGP-1469: Use `/api/v1/index/action/...` instead of `/api/index/action/v1/...` (#57) * DSDEGP-1469: Use `/api/v1/index/action/...` ... instead of `/api/index/action/v1/...` in Clio API. * Sc... — committed to broadinstitute/clio by tbl3rd 7 years ago

Most upvoted comments

In other words, don’t fail fluentd if a log file does not exist to be tracked. Once it does exist, fluentd should fail if it can’t create the log file, as that would be a bad configuration. And when the file winks out of existence again, the pos file should not be touched or considered.

Does that seem reasonable?

portante on Apr 5, 2016

Huh, so if the log file does not exist yet, but will be created shortly, fluentd dies and won’t work. But once the path to the log file is created, then the pos file could be created.

So why not wait until the log file exists and is actually being tracked by fluentd before creating the pos file?

portante on Apr 5, 2016