vector: Journald Source crashes after few minutes

Software Versions

$ vector --version
vector 0.4.0-dev
$ journalctl --version
systemd 241 (241)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid
lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 9.11 (stretch)
Release:	9.11
Codename:	stretch

Vector Config

The following is our vector.toml config file

data_dir = "/var/lib/vector"

#INPPUT 
[sources.journald_myservice]
type = "journald"
units = ["myservice"]

[sources.nginx_access]
type         = "file"
include      = ["/var/log/nginx/access.log"]  
ignore_older = 86400
 
[transforms.journald_myservice_json]
type = "json_parser"
drop_invalid = false
inputs = ["journald_myservice"]

#OUTPUT JOURNALD 
[sinks.journald_es]
type = "elasticsearch"
host = "https://xxxxxxxxxxxxxx"
inputs = ["journald_myservice_json"]
index = "journald-%Y-%m-%d"
batch_size = 10000000
[sinks.journald_es.basic_auth]
user = "xxxxxxx"
password = "xxxxxxxxx"

#OUTPUT NGINX
[sinks.es]
type = "elasticsearch"
host = "https://xxxxxxxxxxxxxx"
inputs = ["nginx_access"]
index = "vector-%Y-%m-%d"
batch_size = 10000000
[sinks.es.basic_auth]
user = "xxxxxxx"
password = "xxxxxxxxx"

Note that myservice is a custom nodejs service, it creates arround 4000 log per minutes

Issue

There appears to be an issue with the journald source, When vector is started everything goes smoothly, few minutes later (2-5 minutes) I stop receiving logs from journald on elasticsearch, but I keep receiving logs from nginx(source=file),

Once I run systemctl restart vector

I start receiving again, then only journald crashes after few minutes,

I tried running in verbose mode, with a console sinks, here’s an extract of the logs the moment that journald source crashes: … … {“_BOOT_ID”:“3726528d4b5a461192a0f971d127e5ae”,"…} Nov 17 11:54:07.540 DEBUG sink{name=“journald_es”}: vector::sinks::util::http: response. status=200 OK version=HTTP/1.1 Nov 17 11:54:08.485 DEBUG sink{name=“journald_es”}: vector::sinks::util::http: sending request. Nov 17 11:54:08.514 DEBUG sink{name=“journald_es”}: vector::sinks::util::http: response. status=200 OK version=HTTP/1.1 Nov 17 11:54:08.629 DEBUG sink{name=“es”}: vector::sinks::util::http: sending request. Nov 17 11:54:08.658 DEBUG sink{name=“es”}: vector::sinks::util::http: response. status=200 OK version=HTTP/1.1 Nov 17 11:54:09.652 DEBUG sink{name=“es”}: vector::sinks::util::http: sending request. Nov 17 11:54:09.686 DEBUG sink{name=“es”}: vector::sinks::util::http: response. status=200 OK

Thanks.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 32 (17 by maintainers)

Most upvoted comments

👍 Solved with

[sinks.logs_elasticsearch.request]
 rate_limit_num = 1000

Yes, apologies for that. Docs have been updated accordingly.

I’m sorry about that @soufiane-bouchaara, it seems like the documentation is off with some of the names of those options for the ES sink. I’ve opened #1262 to track a fix, and you should be able to use the actual option names listed there in the meantime.

This is odd, thanks for reporting @soufiane-bouchaara. @bruceg wrote this source so I’ll have him take a look and see what’s going on here.