victorialogs-datasource: Time Series panel bug, when using quantile() or count() with stats()

Hi, I’m using the next stack and their version: Grafana - v11.2.3 Victoria Logs - v1.12.0 Victoria Logs Grafana Datasource - v0.15.0

When I’m using such query in Grafana Time Series panel namespace:= "ingress-nginx" AND cluster: "eks-prod-eu-west-3" | unpack_json | keep ingress_host,upstream_response_time,path | path:!~ "/files|/content|/blog|/assets" | stats by(path,ingress_host) quantile(0.95, upstream_response_time) p95 | p95:>1 I’m seeing in Grafana just dot’s all the time, instead of graph accross time ranges

Image

The same query but in Vlogs UI seems more workable:

Image

Image

Futhermore values in Grafana tottally differ from values in Vlogs UI. In Vlogs UI they are more true, because logs are corelating with p95 values, but in Grafana they are very low.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 60 (27 by maintainers)

Most upvoted comments

I have modified your query and it works

Interesting, from my side too, BUT when I added keep() it works like in this bug, just dots, why??

Before pipes

Image

When using unpack_json

Image

When using keep operator timestamp field is removed

Image

And all values are returned like simple logs. Plugin behaves as expected.

If it is ok for you, please close the issue.

When you use keep operator it removes _time field, and all fields are returned without a timestamp, which is why you got the response you described

oh, understood… so sorry for such long investigation, so no bug from your side, I appologize

Also dots problem, instead of range graph has even simple count() function namespace:= "ingress-nginx" AND cluster:= "eks-prod-eu-west-3" | unpack_json | country:!"-" AND ingress_host:= "test.com" | keep ingress_host,country | stats by (country) count()

as I see all problems comes from stats()

Just friendly reminder

Image

and Hits type of query must be used also for quantile() queries?

It will work the same way as vmui hits, but in the vmui the chart is constructed in the form of composite bars. In the Grafana you can use any of the graph types. Also you can use Logs panel to show the logs from the previous request

but as I saw, your fix was an adding extra type of query, why in this case my bug will not reproduce, if I will use again Range type of query, according to your plugin documentation?

When the request is going to the VictoriaLogs, it returns stats information without a timestamp by design for your query. Grafana relies on timestamps to show the graphs correctly. Plugin just parses the response from victorialogs and transforms it to the data frames in the Grafana representation. If you want to build graphs like it in the vmui you need to use the hits API, if you want to use stats range query, you need to change your query in the way when Victorialogs returns timestamps for your data.

As you can see in the vmui the same request shows only the hits response and tells from the stats range request that the timestamps are missing. The graph in vmui builds by the response from the hits API.

and your changes fixing that problem, so Grafana understanding that timestamps are present and can build graph?

You can build the both types of the graphs like in the vmui First one can be a bar chart with the hits API response, and the second one can be the logs panel with the stats range response. And it will look like the VMUI panel

But I need such graphs, using LogsQL and stats() with count() or quantile()

Image

To build the graph like time range, the response from the VictoriaLogs itself needs to contain the timestamps, but for your request, it returns only just the values but not timestamps. I need to check how to modify your query or maybe we have a bug inside VictoriaLogs itself

Is Vlogs UI will show the same info as API call? Because in Vlogs UI I haven’t issue, all problems are, when I’m using Grafana datasource plugin

Can make a copy of the response to the vmui in the network tab? And share it here, it would be enough

Attached, is that what you want? response of the query namespace:= "ingress-nginx" AND cluster: "eks-prod-eu-west-3" | unpack_json | keep ingress_host,upstream_response_time,path | path:!~ "/files|/content|/blog|/assets" | stats by(path,ingress_host) quantile(0.95, upstream_response_time) p95 | p95:>1 debug_vlogs.txt

As I can see from the response, there are no any time fields so the plugin just builds the graph without the timeframes, because the stats returns results like that You may need to update the query to get the response with the timeframes

Previously, I already tried to add _time:5m, nothing changed

Could you modify your query like

_time:5m namespace:= "ingress-nginx" AND cluster: "eks-prod-eu-west-3" | unpack_json | keep ingress_host,upstream_response_time,path | path:!~ "/files|/content|/blog|/assets" | stats by(path,ingress_host) quantile(0.95, upstream_response_time) p95 | p95:>1

Does it help? I will check the request for this query. Maybe it makes incorrect params

No, still the same Screenshosts and debug logs attached

debug_vlogs.txt

Image

I see that the vmui uses the hits API response, but when you try to make the request to the range in the plugin, it requests the stats_range API. Need to check logic behind, will be back soon. Thanks for your logs and screens