victorialogs-datasource: incorrect interpolation of multi-value variable in regex
the query: host:~'^$host$' with host variable multi select of backup3 and backup4
is interpolated to: host:~'^"(backup3|backup4)"$' (as seen in the query inspector)
There are additional quotes around the regex.
Expected query: host:~'^(backup3|backup4)$'
Background Use Case:
I would like to filter logs based on multiple selected values, of e.g. the host label
The dashboard variables returns multiple values formatted regex (foo|bar|three).
if using regexp filter host:~“$host” it partially works, but matches anything with that substring. also, regexp filter is not good for performance
Originally, the workaround with using regex start and end matchers worked, but later the quotes were introduced, now this doesn’t work anymore. I would prefer that the quoting is left to the user of the variable and not returning quotes in the variable value. https://github.com/VictoriaMetrics/victorialogs-datasource/issues/65#issuecomment-2312876313
UPDATE: host:in($host) works correctly. Simplified this issue to be specific about the interpolation in regex.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 15 (10 by maintainers)
Commits related to this issue
- lib/logstorage: allow passing `*` at `in(*)`, `contains_any(*)` and `contains_all(*)` Such filters are equivalent to `match all` filter aka `*`. These filters are needed for VictoriaLogs plugin for G... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: allow passing `*` at `in(*)`, `contains_any(*)` and `contains_all(*)` Such filters are equivalent to `match all` filter aka `*`. These filters are needed for VictoriaLogs plugin for G... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: support for `{field in (*)}` and `{field not_in (*)}` syntax in LogsQL This is needed for https://github.com/VictoriaMetrics/victorialogs-datasource/issues/238 to be consistent with `... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: support for `{field in (*)}` and `{field not_in (*)}` syntax in LogsQL This is needed for https://github.com/VictoriaMetrics/victorialogs-datasource/issues/238 to be consistent with `... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: allow passing `*` at `in(*)`, `contains_any(*)` and `contains_all(*)` Such filters are equivalent to `match all` filter aka `*`. These filters are needed for VictoriaLogs plugin for G... — committed to G-Research/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: support for `{field in (*)}` and `{field not_in (*)}` syntax in LogsQL This is needed for https://github.com/VictoriaMetrics/victorialogs-datasource/issues/238 to be consistent with `... — committed to G-Research/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: allow passing `*` at `in(*)`, `contains_any(*)` and `contains_all(*)` Such filters are equivalent to `match all` filter aka `*`. These filters are needed for VictoriaLogs plugin for G... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: support for `{field in (*)}` and `{field not_in (*)}` syntax in LogsQL This is needed for https://github.com/VictoriaMetrics/victorialogs-datasource/issues/238 to be consistent with `... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: allow passing `*` at `in(*)`, `contains_any(*)` and `contains_all(*)` Such filters are equivalent to `match all` filter aka `*`. These filters are needed for VictoriaLogs plugin for G... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: support for `{field in (*)}` and `{field not_in (*)}` syntax in LogsQL This is needed for https://github.com/VictoriaMetrics/victorialogs-datasource/issues/238 to be consistent with `... — committed to truepele/VictoriaMetrics by valyala 2 years ago
and unless a custom value “*” for All is specified, the entire list should be passed, as the list could be filtered in the variables query
@hagen1778 in general, I agree with you. But I would leave the quoting to the user. especially for regex filters.
the user should write
field:~"$field_filter"This allows usecases like
container_name:~"^$project-.*"which would expand tocontainer_name:~"^project1-.*"orcontainer_name:~"^(project1|project2)-.*"or for All:container_name:~"^.*-.*"FYI, the commit https://github.com/VictoriaMetrics/VictoriaMetrics/commit/84d5771b41cef5f8b80acd8c11b8546aadc5b0f7 adds support for
in(*)filter into VictoriaLogs. This filter is treated asmatch all. This commit will be included in the next release of VictoriaLogs. After that the VictoraLogs datasource can replace the$field_filterwithin(*)ifall valueswere selected by the user for the given field.I just noticed, thanks to VictoriaMetrics/VictoriaMetrics#200, that there’s more magic in the interpolation. Is there some documentation about this?
host:in($host)actually works correctly and is resolved tohost:in(backup3,backup4,dev)host:=$hostis interpreted in yet another way, it is transformed tohost:=backup3 OR backup4 OR devwhich is not correct, as reported in VictoriaMetrics/VictoriaMetrics#200