VictoriaLogs: VictoriaLogs returns nothing when limit is < n/2+1
Describe the bug
I’m sending logs to vlogs via the ElasticSearch endpoint from opentelemetry-collector. Metrics indicate that logs are being ingested with no errors, but I don’t see anything through vmui. The histogram indicates that there are entries, but nothing shows in the results list, even for a * query.
After some experimenting, it turns out that if I remove the limit from the query then it returns all entries (around 37000 in my case). Further experimentation shows that if a time range has n entries then the limit must be at least n/2+1 for anything to show up.
To Reproduce
Count number of entries in a time range and set the limit to 2/n or less. Nothing shows up. With a higher limit, data is returned (but in my case that crashes the browser due to the large data set)
Version
victoria-logs-20240710-011043-tags-v0.28.0-victorialogs-0-gb173f24041
Logs
No errors are logged
Screenshots
curl 'https://vlogs/select/logsql/query' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0' -H 'Accept: application/stream+json' -H 'Accept-Language: en-US,en;q=0.5' -H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Priority: u=4' -H 'TE: trailers' --data-raw 'query=*&start=2024-08-07T19%3A23%3A12.281Z&end=2024-08-07T19%3A28%3A12.281Z&limit=40000'|wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 97.0M 0 97.0M 100 87 26.1M 23 0:00:03 0:00:03 --:--:-- 26.1M
37126
curl 'https://vlogs/select/logsql/query' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0' -H 'Accept: application/stream+json' -H 'Accept-Language: en-US,en;q=0.5' -H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Priority: u=4' -H 'TE: trailers' --data-raw 'query=*&start=2024-08-07T19%3A23%3A12.281Z&end=2024-08-07T19%3A28%3A12.281Z&limit=18563'|wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 87 0 0 100 87 0 62 0:00:01 0:00:01 --:--:-- 62
0
curl 'https://vlogs/select/logsql/query' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0' -H 'Accept: application/stream+json' -H 'Accept-Language: en-US,en;q=0.5' -H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Priority: u=4' -H 'TE: trailers' --data-raw 'query=*&start=2024-08-07T19%3A23%3A12.281Z&end=2024-08-07T19%3A28%3A12.281Z&limit=18564'|wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 51.8M 0 51.8M 100 87 21.9M 36 0:00:02 0:00:02 --:--:-- 21.9M
18564
Used command-line flags
- --retentionPeriod=1
- --storageDataPath=/storage
- --envflag.enable=true
- --envflag.prefix=VM_
- --loggerFormat=json
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (17 by maintainers)
Commits related to this issue
- app/vlselect: consistently reuse the original query timestamp when executing /select/logsql/query with positive limit=N query arg Previously the query could return incorrect results, since the query ... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- app/vlselect: consistently reuse the original query timestamp when executing /select/logsql/query with positive limit=N query arg Previously the query could return incorrect results, since the query ... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- app/vlselect/logsql: call `Query.Optimize()` on the cloned query in order to replace `*` filter with `filterNoop` inside getLastNQueryResults() Updates https://github.com/VictoriaMetrics/VictoriaMetr... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- app/vlselect/logsql: call `Query.Optimize()` on the cloned query in order to replace `*` filter with `filterNoop` inside getLastNQueryResults() Updates https://github.com/VictoriaMetrics/VictoriaMetr... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- docs: [VictoriaLogs] OTel Collector elasticsearchexporter header note (#7074) ### Describe Your Changes By default, the `elasticsearchexporter` in OTel Collector puts the log message under a fiel... — committed to VictoriaMetrics/VictoriaMetrics by jiekun 2 years ago
- docs: [VictoriaLogs] OTel Collector elasticsearchexporter header note (#7074) ### Describe Your Changes By default, the `elasticsearchexporter` in OTel Collector puts the log message under a field o... — committed to VictoriaMetrics/VictoriaMetrics by jiekun 2 years ago
- feature: [victorialogs] drop logs without non-empty _msg field (#7056) ### Describe Your Changes VictoriaLogs allows logs without `_msg` field or `_msg` field is empty. This lead to incorrect sea... — committed to VictoriaMetrics/VictoriaMetrics by jiekun 2 years ago
- feature: [victorialogs] drop logs without non-empty _msg field (#7056) ### Describe Your Changes VictoriaLogs allows logs without `_msg` field or `_msg` field is empty. This lead to incorrect sea... — committed to VictoriaMetrics/VictoriaMetrics by jiekun 2 years ago
- app/vlselect: consistently reuse the original query timestamp when executing /select/logsql/query with positive limit=N query arg Previously the query could return incorrect results, since the query ... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- app/vlselect/logsql: call `Query.Optimize()` on the cloned query in order to replace `*` filter with `filterNoop` inside getLastNQueryResults() Updates https://github.com/VictoriaMetrics/VictoriaMetr... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- docs: [VictoriaLogs] OTel Collector elasticsearchexporter header note (#7074) ### Describe Your Changes By default, the `elasticsearchexporter` in OTel Collector puts the log message under a fiel... — committed to truepele/VictoriaMetrics by jiekun 2 years ago
- feature: [victorialogs] drop logs without non-empty _msg field (#7056) ### Describe Your Changes VictoriaLogs allows logs without `_msg` field or `_msg` field is empty. This lead to incorrect sea... — committed to truepele/VictoriaMetrics by jiekun 2 years ago
- app/vlselect: consistently reuse the original query timestamp when executing /select/logsql/query with positive limit=N query arg Previously the query could return incorrect results, since the query ... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- app/vlselect/logsql: call `Query.Optimize()` on the cloned query in order to replace `*` filter with `filterNoop` inside getLastNQueryResults() Updates https://github.com/VictoriaMetrics/VictoriaMetr... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- docs: [VictoriaLogs] OTel Collector elasticsearchexporter header note (#7074) ### Describe Your Changes By default, the `elasticsearchexporter` in OTel Collector puts the log message under a fiel... — committed to truepele/VictoriaMetrics by jiekun 2 years ago
- feature: [victorialogs] drop logs without non-empty _msg field (#7056) ### Describe Your Changes VictoriaLogs allows logs without `_msg` field or `_msg` field is empty. This lead to incorrect sea... — committed to truepele/VictoriaMetrics by jiekun 2 years ago
Hi @jiekun I went through the issue and looking to contribute here, though need some time for more clarification and understanding Wanted to know if I can take up this as my first issue here or if you have any suggestions lmk 😃 Thanks!
FYI, the commit 7185fe012b14bed0c1a9850a2d054e609e22dff8 drops the ingested logs if they do not contain
_msgfield or if this field is empty, since this field is required to be non-empty in VictoriaLogs data model. The dropped log entry is written to VictoriaLogs log withWARN: dropping log line without _msg field; ...message. It also increments thevl_rows_dropped_total{reason="msg_not_exist"}counter, which is exposed at/metricspage, so alerting can be set up on this metric. This should help quickly identifying and fixing the issue with empty_msgfield.This commit will be included in the next release of VictoriaLogs.
After some tests, I believe the main reason is simply due to the lack of documentation on using the OpenTelemetry Collector with elasticsearchexporter. Users did not declare the mapping of the _msg and _time fields in the HTTP Header, which is necessary.
And as suggested, rejects the ingested logs without _msg field and with empty _msg field should be a good approach, which immediately indicated this step and could fix the issue from user side easily.
FYI, the commit e86891b010471c2604ef52f002c352c0ca3f5deb should fix the issue. This commit calls
q.Optimize()on the cloned query inside the binary search loop atgetLastNQueryResults()function. Theq.Optimize()call is needed for converting*filter intofilterNoop. This commit will be included in the next release of VictoriaLogs. In the mean time it is possible to verify the fix by building VictoriaLogs from this commit according to these docs.Note: it is expected that the
*filter matches any logs, even if these logs do not contain_msgfield. This contradicts the general logic for_field_name:*filter, which matches only logs with non-empty_field_namefield, but this exception for the*filter is made for convenience.Also, it is expected that every ingested log entry must contain non-empty
_msgfield. Currently VictoriaLogs incorrectly allows ingesting logs without_msgfield or with empty_msgfield. It would be better rejecting such log entries at ingestion time from consistency and debuggability PoVs. @jiekun , could you prepare a pull request, which rejects the ingested logs without_msgfield and with empty_msgfield (this is the same from VictoriaLogs data model perspective)? VictoriaLogs should expose the counter of the rejected logs at/metricspage, so this counter could be used in alerting. It also should write a sample of rejected logs into its own log in order to simplify debugging of incorrect data ingestion.@jiekun , could you also provide more details to the following statements?
Why VictoriaLogs fails to parse the log in
mapping.mode: rawformat intocolumnsHeader? Does it log parse errors?The referenced
q.Optimize()call at the top ofgetLastNQueryResults()function is executed unconditionally for both cases:limit < n/2+1andlimit >= n/2+1. It is unclear why the*filter is replaced withNoopFilteronly iflimit >= n/2+1.@jiekun , thanks for the investigation of the issue!
Could you provide links to docs about
mapping.mode: rawin the next comment, so future users and/or developers could figure out what is it?If this is some setting, which is frequently used by standard log shippers, then VictoriaLogs needs to be able to properly parse and store logs received from these log shippers. This is a generic rule of thumb for VictoriaMetrics and VictoriaLogs components - they should properly accept data from popular external clients without the need to configure anything at VictoriaMetrics / VictoriaLogs side. This makes more smooth integration with third-party systems.
I was able to reproduce this issue.
By using
mapping.mode: raw, VictoriaLogs failed to parse the log into https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cc2647d212b4124466e23da69a7e19a76f1f67d5/lib/logstorage/block_header.go#L216So the
PrefixFiltercould not work correctly. https://github.com/VictoriaMetrics/VictoriaMetrics/blob/258ccfb9532357af3fdace53ec08cb464b0c5909/lib/logstorage/filter_prefix.go#L57However, when setting
limit >= n/2+1, the query goes to the fast path, where*filter will be optimize into aNoopFilterinstead ofPrefixFilter, so it could retrieve the result: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/657988ac3aabad2d2f6021040633a202587b7bf0/app/vlselect/logsql/logsql.go#L846Please note that the query result is actually incorrect: when use any other search condition instead of
*, it still cannot perform fuzzy match and return the result (because thePrefixFilteris not working)You can reproduce it with my dataset and following command:
victoria-logs-data.zip
And the way to avoid it is:
mapping.mode: raw.@valyala @hagen1778
@jiekun , the commit 657988ac3aabad2d2f6021040633a202587b7bf0 contains some optimizations for the algorithm of finding the time range with the given matching
limit=Nlogs. The optimization adjusts the upper bound of the time range (end) if the current time range has no matching logs.The issue is with the
dhere, when the limit is very small relative to the initial n/2, the algorithm might incorrectly adjust the start and end times, either overshooting or undershooting the correct time range, leading to no results.A check we could add is,
start >= end || d == 0then break the loop