logtrail: Missing logentries

Hi! First of all, thank you for this wonderful plugin. We currently ship logs to both ES and Papertrail via Logstash to be able to both tail and analyze our logs. But we hope to change that to ES only with this plugin.

However, I’ve noticed that LogTrail seems to drop/not show some logs.

In papertrail we have the following logentries:

Bamboo-host bamboo:  2017-01-12 16:00:30,966 WARN [ActiveMQ Transport: tcp:///xxxxx:51553@54663] [Transport] Transport Connection to: tcp://xxxxx:51553 failed: java.io.EOFException
Bamboo-host bamboo:  2017-01-12 16:00:34,472 INFO [http-nio-8085-exec-15] [AccessLogFilter] xxxxx GET https://xxxxx/userlogin!doDefault.action 233579kb
Bamboo-host bamboo:  2017-01-12 16:00:34,778 INFO [http-nio-8085-exec-12] [DefaultAuthenticator] login : '' could not be authenticated with the given password
Bamboo-host bamboo:  2017-01-12 16:00:34,778 WARN [http-nio-8085-exec-12] [DefaultAuthenticator] login : '' tried to login but they do not have USE permission or weren't found. Deleting remember me cookie.
Bamboo-host bamboo:  2017-01-12 16:00:34,778 INFO [http-nio-8085-exec-12] [BambooAuthenticator] Finished doing authentication checks for , took 744.3 ?s
Bamboo-host bamboo:  2017-01-12 16:00:34,778 INFO [http-nio-8085-exec-12] [AccessLogFilter] xxxxx POST https://xxxxx/userlogin.action 226212kb
Bamboo-host bamboo:  2017-01-12 16:00:35,180 INFO [http-nio-8085-exec-13] [AccessLogFilter] xxxxx GET https://xxxxx/rest/pref/latest/user/?expand=preferences.preference&_=1484236794961 219098kb
Bamboo-host bamboo:  2017-01-12 16:00:35,240 INFO [http-nio-8085-exec-20] [AccessLogFilter] xxxxx GET https://xxxxx/rest/menu/latest/appswitcher?_=1484236794962 218850kb

In LogTrail we have the following logentries:

Bamboo-host bamboo:  2017-01-12 16:00:30,966 WARN [ActiveMQ Transport: tcp:///xxxxx:51553@54663] [Transport] Transport Connection to: tcp://xxxxx:51553 failed: java.io.EOFException
Bamboo-host bamboo:  2017-01-12 16:00:34,778 INFO [http-nio-8085-exec-12] [BambooAuthenticator] Finished doing authentication checks for , took 744.3 ?s
Bamboo-host bamboo:  2017-01-12 16:00:34,778 INFO [http-nio-8085-exec-12] [DefaultAuthenticator] login : '' could not be authenticated with the given password
Bamboo-host bamboo:  2017-01-12 16:00:35,240 INFO [http-nio-8085-exec-20] [AccessLogFilter] xxxxx GET https://xxxxxx/rest/menu/latest/appswitcher?_=1484236794962 218850kb

As you can see, the first and last logs are the same, but the logs in between are different, where Papertrail is “the right one”. I can see all the logs in Kibana, but LogTrail seems to miss them. We use the logtrail-4.x-0.1.5 version and unfortunately we’re stuck with Kibana 4.5 until AWS updates their Elasticsearch service.

Our Logtrail conf:

  "tail_interval_in_seconds": 3,
  "max_buckets": 500,
  "nested_objects" : false,
  "default_time_range_in_days" : 0,

I have a feeling that this problem is due too our low refresh rate and when we refresh, aka fetch new logs from ES, the logs we’re missing are not quite ready in ES yet, like they’re still being indexed or similar. And I guess that Logtrail only checks for logs with a higher timestamp than the last Logtrail entry every time it’s refreshing for new logs? Afaik Kibana for example does not do this, Kibana refreshes the whole search interval everytime, so even if a document took a longer time to index, it would show up the next refresh.