fluentd-kubernetes-daemonset: fluent can not parse multiline correctly

image version: fluent/fluentd-kubernetes-daemonset:v0.12-debian docker version: 1.12.5 plugin:

# gem list|grep fluent-plugin
fluent-plugin-kubernetes_metadata_filter (0.27.0)
fluent-plugin-multiline-parser (0.1.1)
fluent-plugin-record-reformer (0.9.0)
fluent-plugin-redis (0.2.3)
fluent-plugin-secure-forward (0.4.5)
fluent-plugin-systemd (0.0.8)

after deploying this image via kubernetes daemonsets,i found parser cannot deal with multiline log.

java trancestack log as follows:

2017-08-10 15:12:11.994 [com.pdcss.sfjc.db.DBPool:55] ERROR com.pdcss.sfjc.db.DBPool - Network error IOException: Connection timed out: connect
  java.sql.SQLException: Network error IOException: Connection timed out: connect
   at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:421)
   at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)
   at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:185)

the filter conf like this:

<filter container.error>
      @type parser
      format multiline
      format_firstline /\d{4}-\d{1,2}-\d{1,2}/
      format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}\.\d{0,3}) \[(?<thread>.*)\] (?<level>[^\s]+) (?<classmathod>[^\s]+) - (?<message>[\s\S]*)/
      key_name all
</filter>

docker log:

2017-08-23 15:29:00 +0800 [warn]: pattern not match with data '  java.sql.SQLException: Network error IOException: Connection timed out: connect'
2017-08-23 15:29:00 +0800 [warn]: pattern not match with data '   at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:421)'
2017-08-23 15:29:00 +0800 [warn]: pattern not match with data '   at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)'
2017-08-23 15:29:00 +0800 [warn]: pattern not match with data '   at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:185)'
2017-08-10 15:28:11 +0800 container.error: {"thread":"com.pdcss.sfjc.db.DBPool:55","level":"ERROR","classmathod":"com.pdcss.sfjc.db.DBPool","message":"Network error IOException: Connection timed out: connect"}

as context log said,filter can parser single line log correctly,but incorrectly when parsing multiline such as tracestack. i guess the format param is wrong with the line break(\n) but no idea about this config. by the way,as documentation fluent multiline parser mentioned,format (?<message>.*) also not work. Any solution on this issue,please?

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 4
  • Comments: 16 (1 by maintainers)

Most upvoted comments

@raf-d Would you mind sharing the configuration you ended up with? I’ve been going down the same path as well.

I know this is a year old, but this is what we ended up doing. Works perfectly and you can join multiple multiline formats with a | symbol. Ex: A|B|C etc… Now we have full java stacktraces in a single event.

<filter **>
  @type concat
  key log
  multiline_start_regexp /\w+ \d+.\d+.\d+ \d+:\d+:\d+: /
  stream_identity_key tag
</filter>

Note: Rancher is going the path of adding official concat support —> https://github.com/rancher/fluentd/pull/15

@raf-d Would you mind sharing the configuration you ended up with? I’ve been going down the same path as well.