fluentd-kubernetes-daemonset: fluent can not parse multiline correctly
image version: fluent/fluentd-kubernetes-daemonset:v0.12-debian docker version: 1.12.5 plugin:
# gem list|grep fluent-plugin
fluent-plugin-kubernetes_metadata_filter (0.27.0)
fluent-plugin-multiline-parser (0.1.1)
fluent-plugin-record-reformer (0.9.0)
fluent-plugin-redis (0.2.3)
fluent-plugin-secure-forward (0.4.5)
fluent-plugin-systemd (0.0.8)
after deploying this image via kubernetes daemonsets,i found parser cannot deal with multiline log.
java trancestack log as follows:
2017-08-10 15:12:11.994 [com.pdcss.sfjc.db.DBPool:55] ERROR com.pdcss.sfjc.db.DBPool - Network error IOException: Connection timed out: connect
java.sql.SQLException: Network error IOException: Connection timed out: connect
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:421)
at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:185)
the filter conf like this:
<filter container.error>
@type parser
format multiline
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}\.\d{0,3}) \[(?<thread>.*)\] (?<level>[^\s]+) (?<classmathod>[^\s]+) - (?<message>[\s\S]*)/
key_name all
</filter>
docker log:
2017-08-23 15:29:00 +0800 [warn]: pattern not match with data ' java.sql.SQLException: Network error IOException: Connection timed out: connect'
2017-08-23 15:29:00 +0800 [warn]: pattern not match with data ' at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:421)'
2017-08-23 15:29:00 +0800 [warn]: pattern not match with data ' at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)'
2017-08-23 15:29:00 +0800 [warn]: pattern not match with data ' at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:185)'
2017-08-10 15:28:11 +0800 container.error: {"thread":"com.pdcss.sfjc.db.DBPool:55","level":"ERROR","classmathod":"com.pdcss.sfjc.db.DBPool","message":"Network error IOException: Connection timed out: connect"}
as context log said,filter can parser single line log correctly,but incorrectly when parsing multiline such as tracestack.
i guess the format param is wrong with the line break(\n) but no idea about this config.
by the way,as documentation fluent multiline parser mentioned,format (?<message>.*)
also not work.
Any solution on this issue,please?
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 4
- Comments: 16 (1 by maintainers)
https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions ?
I know this is a year old, but this is what we ended up doing. Works perfectly and you can join multiple multiline formats with a | symbol. Ex: A|B|C etc… Now we have full java stacktraces in a single event.
Note: Rancher is going the path of adding official concat support —> https://github.com/rancher/fluentd/pull/15
@raf-d Would you mind sharing the configuration you ended up with? I’ve been going down the same path as well.