logagent-js: Logagent stops receiving log lines when monitored container restarts

My logagents repeatedly stop receiving log lines from containers after running for a couple of hours. It seems to be linked to when the monitored containers restart.

Here’s my docker-compose config:

logagent:
  image: sematext/logagent:latest
  environment:
    - LOGS_RECEIVER_URL=https://logsene-receiver.eu.sematext.com
    - LOGSENE_ENABLED_DEFAULT=false
    - LOGSENE_TOKEN=${LOGS_TOKEN_DEFAULT}
    - LOGAGENT_ARGS=--verbose
  restart: always
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock

And my monitored containers have the following env vars:

environment:
  - LOGSENE_ENABLED=true
  - LOGSENE_TOKEN=${LOGS_TOKEN_WORKER}

Here is the log output. Everything is fine (X lines parsed) until the monitored container restarts, after which everything is 0…

2019-05-21T11:05:41.883Z pid[6] Memory stats:  heapUsedMB=18 heapTotalMB=24 memoryRssMB=114
2019-05-21T11:06:41.882Z pid[6] Logagent report: pid[6] 59999 ms 59 lines parsed.  1 lines/s 0.000 MB/s - empty lines: 0
2019-05-21T11:06:41.882Z pid[6] Logagent stats: usedTokens=1 shippedLogs=61 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=1
2019-05-21T11:06:41.882Z pid[6] Memory stats:  heapUsedMB=18 heapTotalMB=24 memoryRssMB=114
2019-05-21T11:07:41.882Z pid[6] Logagent report: pid[6] 60001 ms 40 lines parsed.  1 lines/s 0.000 MB/s - empty lines: 0
2019-05-21T11:07:41.883Z pid[6] Logagent stats: usedTokens=1 shippedLogs=37 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=1
2019-05-21T11:07:41.883Z pid[6] Memory stats:  heapUsedMB=18 heapTotalMB=24 memoryRssMB=114
2019-05-21T11:08:41.883Z pid[6] Logagent report: pid[6] 60001 ms 55 lines parsed.  1 lines/s 0.000 MB/s - empty lines: 0
2019-05-21T11:08:41.884Z pid[6] Logagent stats: usedTokens=1 shippedLogs=59 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=1
2019-05-21T11:08:41.884Z pid[6] Memory stats:  heapUsedMB=16 heapTotalMB=24 memoryRssMB=114
2019-05-21T11:09:41.883Z pid[6] Logagent report: pid[6] 60000 ms 52 lines parsed.  1 lines/s 0.000 MB/s - empty lines: 0
2019-05-21T11:09:41.884Z pid[6] Logagent stats: usedTokens=1 shippedLogs=52 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=1
2019-05-21T11:09:41.884Z pid[6] Memory stats:  heapUsedMB=16 heapTotalMB=24 memoryRssMB=114
2019-05-21T11:10:41.883Z pid[6] Logagent report: pid[6] 60000 ms 42 lines parsed.  1 lines/s 0.000 MB/s - empty lines: 0
2019-05-21T11:10:41.884Z pid[6] Logagent stats: usedTokens=1 shippedLogs=35 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=1
2019-05-21T11:10:41.884Z pid[6] Memory stats:  heapUsedMB=17 heapTotalMB=24 memoryRssMB=114
2019-05-21T11:11:41.882Z pid[6] Logagent report: pid[6] 59999 ms 60 lines parsed.  1 lines/s 0.000 MB/s - empty lines: 0
2019-05-21T11:11:41.883Z pid[6] Logagent stats: usedTokens=1 shippedLogs=62 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=1
2019-05-21T11:11:41.884Z pid[6] Memory stats:  heapUsedMB=17 heapTotalMB=25 memoryRssMB=114
2019-05-21T11:12:41.883Z pid[6] Logagent report: pid[6] 60001 ms 16 lines parsed.  0 lines/s 0.000 MB/s - empty lines: 0
2019-05-21T11:12:41.884Z pid[6] Logagent stats: usedTokens=1 shippedLogs=16 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=0
2019-05-21T11:12:41.884Z pid[6] Memory stats:  heapUsedMB=17 heapTotalMB=25 memoryRssMB=113
2019-05-21T11:13:09.591Z pid[6] Container a24e888d660df941da452037242f507ebaddcd4cdc5aa8f5b87abed71dc3f33d /my_worker_4 setting LOGSENE_ENABLED=true
2019-05-21T11:13:41.884Z pid[6] Logagent report: pid[6] 60001 ms 5 lines parsed.  0 lines/s 0.000 MB/s - empty lines: 0
2019-05-21T11:13:41.885Z pid[6] Logagent stats: usedTokens=1 shippedLogs=14 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=0
2019-05-21T11:13:41.885Z pid[6] Memory stats:  heapUsedMB=17 heapTotalMB=25 memoryRssMB=113
2019-05-21T11:14:41.884Z pid[6] Logagent report: pid[6] 60000 ms 0 lines parsed.  0 lines/s 0.000 MB/s - empty lines: 0
2019-05-21T11:14:41.885Z pid[6] Logagent stats: usedTokens=1 shippedLogs=0 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=0
2019-05-21T11:14:41.885Z pid[6] Memory stats:  heapUsedMB=17 heapTotalMB=25 memoryRssMB=114
2019-05-21T11:15:41.885Z pid[6] Logagent report: pid[6] 60000 ms 0 lines parsed.  0 lines/s 0.000 MB/s - empty lines: 0
2019-05-21T11:15:41.885Z pid[6] Logagent stats: usedTokens=1 shippedLogs=0 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=0
2019-05-21T11:15:41.885Z pid[6] Memory stats:  heapUsedMB=17 heapTotalMB=25 memoryRssMB=114
2019-05-21T11:16:41.885Z pid[6] Logagent report: pid[6] 60001 ms 0 lines parsed.  0 lines/s 0.000 MB/s - empty lines: 0
2019-05-21T11:16:41.886Z pid[6] Logagent stats: usedTokens=1 shippedLogs=0 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=0
2019-05-21T11:16:41.886Z pid[6] Memory stats:  heapUsedMB=17 heapTotalMB=25 memoryRssMB=114
2019-05-21T11:17:41.885Z pid[6] Logagent report: pid[6] 60000 ms 0 lines parsed.  0 lines/s 0.000 MB/s - empty lines: 0
2019-05-21T11:17:41.886Z pid[6] Logagent stats: usedTokens=1 shippedLogs=0 httpFailed=0 httpRetransmit=0 throughputLinesPerSecond=0

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

The fileds have a new structure to correlate better with container monitoring labels. Very soon you will be able to navigate by those fields between metrics and logs.

os.host os.host_ip container.name container.image container.id

Kubernetes properties are moved to Kubernetes.pod.name etc

FYI instructions in US and EU regions now show just the region param (no receiver URLs). Of course, make sure you refresh the browser tab where your app is loaded.