VictoriaLogs: VictoriaLogs returns nothing when limit is < n/2+1

Describe the bug

I’m sending logs to vlogs via the ElasticSearch endpoint from opentelemetry-collector. Metrics indicate that logs are being ingested with no errors, but I don’t see anything through vmui. The histogram indicates that there are entries, but nothing shows in the results list, even for a * query.

After some experimenting, it turns out that if I remove the limit from the query then it returns all entries (around 37000 in my case). Further experimentation shows that if a time range has n entries then the limit must be at least n/2+1 for anything to show up.

To Reproduce

Count number of entries in a time range and set the limit to 2/n or less. Nothing shows up. With a higher limit, data is returned (but in my case that crashes the browser due to the large data set)

Version

victoria-logs-20240710-011043-tags-v0.28.0-victorialogs-0-gb173f24041

Logs

No errors are logged

Screenshots

curl 'https://vlogs/select/logsql/query' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0' -H 'Accept: application/stream+json' -H 'Accept-Language: en-US,en;q=0.5' -H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8'  -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Priority: u=4' -H 'TE: trailers' --data-raw 'query=*&start=2024-08-07T19%3A23%3A12.281Z&end=2024-08-07T19%3A28%3A12.281Z&limit=40000'|wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 97.0M    0 97.0M  100    87  26.1M     23  0:00:03  0:00:03 --:--:-- 26.1M
37126


curl 'https://vlogs/select/logsql/query' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0' -H 'Accept: application/stream+json' -H 'Accept-Language: en-US,en;q=0.5'  -H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8'  -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Priority: u=4' -H 'TE: trailers' --data-raw 'query=*&start=2024-08-07T19%3A23%3A12.281Z&end=2024-08-07T19%3A28%3A12.281Z&limit=18563'|wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    87    0     0  100    87      0     62  0:00:01  0:00:01 --:--:--    62
0


curl 'https://vlogs/select/logsql/query' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0' -H 'Accept: application/stream+json' -H 'Accept-Language: en-US,en;q=0.5'  -H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8'  -H 'Connection: keep-alive'  -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Priority: u=4' -H 'TE: trailers' --data-raw 'query=*&start=2024-08-07T19%3A23%3A12.281Z&end=2024-08-07T19%3A28%3A12.281Z&limit=18564'|wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 51.8M    0 51.8M  100    87  21.9M     36  0:00:02  0:00:02 --:--:-- 21.9M
18564

Used command-line flags

    - --retentionPeriod=1                                                                                                                                               
    - --storageDataPath=/storage                                                                                                                                        
    - --envflag.enable=true                                                                                                                                             
    - --envflag.prefix=VM_                                                                                                                                              
    - --loggerFormat=json      

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (17 by maintainers)

Commits related to this issue

Most upvoted comments

Hi @jiekun I went through the issue and looking to contribute here, though need some time for more clarification and understanding Wanted to know if I can take up this as my first issue here or if you have any suggestions lmk 😃 Thanks!

FYI, the commit 7185fe012b14bed0c1a9850a2d054e609e22dff8 drops the ingested logs if they do not contain _msg field or if this field is empty, since this field is required to be non-empty in VictoriaLogs data model. The dropped log entry is written to VictoriaLogs log with WARN: dropping log line without _msg field; ... message. It also increments the vl_rows_dropped_total{reason="msg_not_exist"} counter, which is exposed at /metrics page, so alerting can be set up on this metric. This should help quickly identifying and fixing the issue with empty _msg field.

This commit will be included in the next release of VictoriaLogs.

Why VictoriaLogs fails to parse the log in mapping.mode: raw format into columnsHeader? Does it log parse errors?

After some tests, I believe the main reason is simply due to the lack of documentation on using the OpenTelemetry Collector with elasticsearchexporter. Users did not declare the mapping of the _msg and _time fields in the HTTP Header, which is necessary.

And as suggested, rejects the ingested logs without _msg field and with empty _msg field should be a good approach, which immediately indicated this step and could fix the issue from user side easily.

FYI, the commit e86891b010471c2604ef52f002c352c0ca3f5deb should fix the issue. This commit calls q.Optimize() on the cloned query inside the binary search loop at getLastNQueryResults() function. The q.Optimize() call is needed for converting * filter into filterNoop. This commit will be included in the next release of VictoriaLogs. In the mean time it is possible to verify the fix by building VictoriaLogs from this commit according to these docs.

Note: it is expected that the * filter matches any logs, even if these logs do not contain _msg field. This contradicts the general logic for _field_name:* filter, which matches only logs with non-empty _field_name field, but this exception for the * filter is made for convenience.

Also, it is expected that every ingested log entry must contain non-empty _msg field. Currently VictoriaLogs incorrectly allows ingesting logs without _msg field or with empty _msg field. It would be better rejecting such log entries at ingestion time from consistency and debuggability PoVs. @jiekun , could you prepare a pull request, which rejects the ingested logs without _msg field and with empty _msg field (this is the same from VictoriaLogs data model perspective)? VictoriaLogs should expose the counter of the rejected logs at /metrics page, so this counter could be used in alerting. It also should write a sample of rejected logs into its own log in order to simplify debugging of incorrect data ingestion.

@jiekun , could you also provide more details to the following statements?

By using mapping.mode: raw, VictoriaLogs failed to parse the log into columnsHeader … so the PrefixFilter could not work correctly.

Why VictoriaLogs fails to parse the log in mapping.mode: raw format into columnsHeader? Does it log parse errors?

However, when setting limit >= n/2+1, the query goes to the fast path, where * filter will be optimize into a NoopFilter instead of PrefixFilter, so it could retrieve the result.

The referenced q.Optimize() call at the top of getLastNQueryResults() function is executed unconditionally for both cases: limit < n/2+1 and limit >= n/2+1. It is unclear why the * filter is replaced with NoopFilter only if limit >= n/2+1.

@jiekun , thanks for the investigation of the issue!

Could you provide links to docs about mapping.mode: raw in the next comment, so future users and/or developers could figure out what is it?

If this is some setting, which is frequently used by standard log shippers, then VictoriaLogs needs to be able to properly parse and store logs received from these log shippers. This is a generic rule of thumb for VictoriaMetrics and VictoriaLogs components - they should properly accept data from popular external clients without the need to configure anything at VictoriaMetrics / VictoriaLogs side. This makes more smooth integration with third-party systems.

I was able to reproduce this issue.

By using mapping.mode: raw, VictoriaLogs failed to parse the log into https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cc2647d212b4124466e23da69a7e19a76f1f67d5/lib/logstorage/block_header.go#L216

So the PrefixFilter could not work correctly. https://github.com/VictoriaMetrics/VictoriaMetrics/blob/258ccfb9532357af3fdace53ec08cb464b0c5909/lib/logstorage/filter_prefix.go#L57

However, when setting limit >= n/2+1, the query goes to the fast path, where * filter will be optimize into a NoopFilter instead of PrefixFilter, so it could retrieve the result: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/657988ac3aabad2d2f6021040633a202587b7bf0/app/vlselect/logsql/logsql.go#L846

Please note that the query result is actually incorrect: when use any other search condition instead of *, it still cannot perform fuzzy match and return the result (because the PrefixFilter is not working)

You can reproduce it with my dataset and following command:

victoria-logs-data.zip

curl 'http://127.0.0.1:9428/select/logsql/query' \
  -H 'Accept: application/stream+json' \
  -H 'Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7' \
  -H 'AccountID: 0' \
  -H 'ProjectID: 0' \
  --data-raw 'query=*&limit=2&start=2024-09-18T11%3A55%3A35.698Z&end=2024-09-18T12%3A00%3A35.698Z'

And the way to avoid it is:

  1. Do not use mapping.mode: raw.
  2. Enhance VictoriaLogs elasticsearch ingest API to handle data in this format.

@valyala @hagen1778

@jiekun , the commit 657988ac3aabad2d2f6021040633a202587b7bf0 contains some optimizations for the algorithm of finding the time range with the given matching limit=N logs. The optimization adjusts the upper bound of the time range (end) if the current time range has no matching logs.

The issue is with the d here, when the limit is very small relative to the initial n/2, the algorithm might incorrectly adjust the start and end times, either overshooting or undershooting the correct time range, leading to no results.

A check we could add is, start >= end || d == 0 then break the loop