VictoriaLogs: VictoriaLogs Cluster: missing _msg field for any ingest API
Is your question request related to a specific component?
vector config
[sources.kubernetes_logs]
[transforms.parse_logs]
type = "remap"
inputs = ["kubernetes_logs"]
source = """
json_parsed, err = parse_json(.message)
if err == null && json_parsed != null {
.= merge!(., json_parsed)
} else {
nginx_log, err1 = parse_nginx_log(.message, "combined")
if err1 == null && nginx_log != null {
.= merge(., nginx_log)
}
}
.app = .kubernetes.pod_labels.app
.container_name = .kubernetes.container_name
.pod_name = .kubernetes.pod_name
.namespace = .kubernetes.pod_namespace
log_level, err = get(., ["level"])
if err == null && log_level != null {
. = set!(., ["level"], log_level)
} else {
. = set!(., ["level"], "info")
}
del(.kubernetes)
del(.file)
del(.cost)
del(.stream)
del(.source_type)
"""
[sinks.vmlog]
type = "loki"
inputs = ["parse_logs"]
endpoint = "http://vlinsert.ops.svc:9481/insert/loki/"
path = "/api/v1/push?_stream_fields=app,pod_name,container_name,namespace,level&_msg_field=message&_time_field=timestamp&disable_message_parsing=1"
compression = "none"
out_of_order_action = "accept"
remove_label_fields = true
tenant_id = "1"
[sinks.vmlog.request]
concurrency = 10
rate_limit_num = 500
retry_attempts = 3
[sinks.vmlog.healthcheck]
enabled = false
[sinks.vmlog.labels]
"app" = "{{ .app }}"
"container_name" = "{{ .container_name }}"
"pod_name" = "{{ .pod_name }}"
"namespace" = "{{ .namespace }}"
"level" = "{{ .level }}"
[sinks.vmlog.encoding]
codec = "raw_message"
vlstorage
vminsert
Describe the question in detail
I migrated from loki to VictoriaLogs, why does it prompt me that I don’t have the msg field?
Troubleshooting docs
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 27 (14 by maintainers)
Commits related to this issue
- lib/logstorage: expect message in a field with empty and _msg name (#8743) ### Describe Your Changes fixes #8707 ### Checklist The following checks are **mandatory**: - [ ] My change adheres to [... — committed to VictoriaMetrics/VictoriaMetrics by AndrewChubatiuk 2 years ago
- docs/victorialogs/CHANGELOG.md: clarify the description of the bugfix in the commit 0fee22e91a8cab2fe34232510eef5ffc4ccd12b7 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8743 Updat... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- docs/victorialogs/CHANGELOG.md: clarify the description of the bugfix in the commit 0fee22e91a8cab2fe34232510eef5ffc4ccd12b7 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8743 Updat... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: expect message in a field with empty and _msg name (#8743) ### Describe Your Changes fixes #8707 ### Checklist The following checks are **mandatory**: - [ ] My change adheres to [... — committed to MorningSong/VictoriaMetrics by AndrewChubatiuk 2 years ago
- lib/logstorage: expect message in a field with empty and _msg name (#8743) ### Describe Your Changes fixes #8707 ### Checklist The following checks are **mandatory**: - [ ] My change adheres to [... — committed to truepele/VictoriaMetrics by AndrewChubatiuk 2 years ago
- docs/victorialogs/CHANGELOG.md: clarify the description of the bugfix in the commit 0fee22e91a8cab2fe34232510eef5ffc4ccd12b7 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8743 Updat... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: expect message in a field with empty and _msg name (#8743) ### Describe Your Changes fixes #8707 ### Checklist The following checks are **mandatory**: - [ ] My change adheres to [... — committed to truepele/VictoriaMetrics by AndrewChubatiuk 2 years ago
- docs/victorialogs/CHANGELOG.md: clarify the description of the bugfix in the commit 0fee22e91a8cab2fe34232510eef5ffc4ccd12b7 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/8743 Updat... — committed to truepele/VictoriaMetrics by valyala 2 years ago
Hi,
I have the same problem with the syslog input on version “v1.18.0-victorialogs” In a single node variant everything works as expected. In a cluster consisting of vlinsert/vlselect + vlstorage i get the “missing _msg field” error.
Here’s a log from the vlstorage node:
The vlinsert/vlselect node is started using the following command, but doesn’t output any logs.
Hi. This is my configuration. I use vector to collect the logs of a service and write them to both v1.15.0 and v1.18.0 clusters. I don’t know why the cluster version reports missing the _msg field.
Hi. Here is some vector console sink data
vector_logs.txt