telegraf: Tail input plugin does not read line-by-line and sends partial line data

Relevant telegraf.conf

# Configuration for telegraf agent
[agent]
    interval = "60s"
    debug = false
    hostname = "<hostname>"
    round_interval = true
    flush_interval = "60s"
    flush_jitter = "0s"
    collection_jitter = "0s"
    metric_batch_size = 3000
    metric_buffer_limit = 15000
    quiet = false
    logtarget = "file"
    logfile = "/opt/logs/telegraf.log"
    logfile_rotation_interval = "24h"
    logfile_rotation_max_size = "10MB"
    logfile_rotation_max_archives = 5
    omit_hostname = false

[[outputs.influxdb]]
    urls = ["https://<influxdb>:8085"]
    database = "monitoring"
    precision = "60s"
    insecure_skip_verify = true

[[inputs.tail]]
    files = ["/path/to/file/telegraf-metrics.log*"]
    data_format = "influx"
    path_tag = ""
[inputs.tail.tags]
    metric_type = "manager"

Logs from Telegraf

2022-06-22T19:26:00Z E! [inputs.tail] Malformed log line in "/path/to/file/telegraf-metrics.log_2022-06-22-15-22": ["e.partitioncount=2048,offheap.backupcount=27,offheap.entrycount=36,offheap.primarycount=9,offheap.size=0,onheap.entrycount=0,put.rate=1111,rebalance.keyleft=0,rebalance.keyrate=0,rebalance.partitioncount=0,tx.commit.size=0,tx.commit.time=141250,tx.rollback.size=0 1655925960896000000"]: metric parse error: expected field at 1:284: "e.partitioncount=2048,offheap.backupcount=27,offheap.entrycount=36,offheap.primarycount=9,offheap.size=0,onheap.entrycount=0,put.rate=1111,rebalance.keyleft=0,rebalance.keyrate=0,rebalance.partitioncount=0,tx.commit.size=0,tx.commit.time=141250,tx.rollback.size=0 1655925960896000000"

System info

Telegraf 1.18.3 (git: HEAD 6a94f65a), RHEL 7.9 (Maipo) 63.10.0-1160.31.1.el7.x86_64 and RHEL 6.10 (Santiago) 2.6.32-754.35.1.el6.x86_64

Docker

No response

Steps to reproduce

  1. Application writes out InfluxDB line protocol formatted metrics to file.
  2. Tail plugin reads and publishes to configured InfluxDB output.
  3. Malformed lines messages in logs and bad measurements in InfluxDB.

Expected behavior

Full lines should be sent instead of partially reading lines and sending them.

Actual behavior

Lines are partially being read and sent leading to large number of bad measurements in InfluxDB.

Additional info

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 19 (10 by maintainers)

Most upvoted comments

I will have to pick up some Go 😔