telegraf: Tail input plugin does not read line-by-line and sends partial line data
Relevant telegraf.conf
# Configuration for telegraf agent
[agent]
interval = "60s"
debug = false
hostname = "<hostname>"
round_interval = true
flush_interval = "60s"
flush_jitter = "0s"
collection_jitter = "0s"
metric_batch_size = 3000
metric_buffer_limit = 15000
quiet = false
logtarget = "file"
logfile = "/opt/logs/telegraf.log"
logfile_rotation_interval = "24h"
logfile_rotation_max_size = "10MB"
logfile_rotation_max_archives = 5
omit_hostname = false
[[outputs.influxdb]]
urls = ["https://<influxdb>:8085"]
database = "monitoring"
precision = "60s"
insecure_skip_verify = true
[[inputs.tail]]
files = ["/path/to/file/telegraf-metrics.log*"]
data_format = "influx"
path_tag = ""
[inputs.tail.tags]
metric_type = "manager"
Logs from Telegraf
2022-06-22T19:26:00Z E! [inputs.tail] Malformed log line in "/path/to/file/telegraf-metrics.log_2022-06-22-15-22": ["e.partitioncount=2048,offheap.backupcount=27,offheap.entrycount=36,offheap.primarycount=9,offheap.size=0,onheap.entrycount=0,put.rate=1111,rebalance.keyleft=0,rebalance.keyrate=0,rebalance.partitioncount=0,tx.commit.size=0,tx.commit.time=141250,tx.rollback.size=0 1655925960896000000"]: metric parse error: expected field at 1:284: "e.partitioncount=2048,offheap.backupcount=27,offheap.entrycount=36,offheap.primarycount=9,offheap.size=0,onheap.entrycount=0,put.rate=1111,rebalance.keyleft=0,rebalance.keyrate=0,rebalance.partitioncount=0,tx.commit.size=0,tx.commit.time=141250,tx.rollback.size=0 1655925960896000000"
System info
Telegraf 1.18.3 (git: HEAD 6a94f65a), RHEL 7.9 (Maipo) 63.10.0-1160.31.1.el7.x86_64 and RHEL 6.10 (Santiago) 2.6.32-754.35.1.el6.x86_64
Docker
No response
Steps to reproduce
- Application writes out InfluxDB line protocol formatted metrics to file.
- Tail plugin reads and publishes to configured InfluxDB output.
- Malformed lines messages in logs and bad measurements in InfluxDB.
Expected behavior
Full lines should be sent instead of partially reading lines and sending them.
Actual behavior
Lines are partially being read and sent leading to large number of bad measurements in InfluxDB.
Additional info
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 19 (10 by maintainers)
I will have to pick up some Go 😔