prometheus-engine: PromQL for Google Cloud Monitoring metrics doesn't support label regexs for integers
In Prometheus, all metric labels are strings, and PromQL supports label selection with regular expressions. In Cloud Monitoring, some metric labels are integers.
For example, we use a log based metric that counts HTTP requests by status code. The status code is INT64 label for the metric. Trying to query this metric using PromQL, we get the following error:
Error executing query: invalid parameter "query": substitute queries failed: convert vector selector failed: unable to parse value in matcher on int label to an int: strconv.ParseInt: parsing "5\\d\\d": invalid syntax.
The query is trying to count up the 5xx status codes, for use in an alert. It appears as if the label is an integer type, we can no longer use the label regex in PromQL.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 17
- Comments: 30 (2 by maintainers)
PromQL replatforming is code complete, it will roll out this month. This is the first feature that we’ll deliver once that’s done. Thanks for keeping the pressure on 😃
Yeah - we’re super close to finishing the replatforming of PromQL, should be ready this month. At that point we’ll switch everyone over and this will work. 😃
Adding >/< operators to PromQL would break spec with PromQL (as all label values in PromQL must be strings) and we really don’t want to do that. The real problem is that we even have INT value labels in the first place for GCP metrics. IMO the real solution is for us to be able to treat INT values like Strings, and then you can do whatever with PromQL and we don’t break compatibility.
Thanks for nudging me - we’re still working on this but unfortunately we’re not hitting the September date 😕
ETA early 2025, hopefully. It’s a top priority of ours.
Just a heads up that while using label_replace() might work, using it inside an aggregation function is likely to move that part of the query execution out of the very efficient Monarch query engine and into the much less efficient PromQL query engine. In other words, while what you’re doing might work just fine for small amounts of data, it is likely to time out or cause other errors for large amounts of data.
Nothing yet but this work is prioritized as part of a larger promql refactoring happening soon. I’d expect it to be unblocked sometime around September.
Code is done, but we’re in a big freeze because of the large GCP-wide outage a few weeks back. It’ll be rolled out as soon as that’s lifted.
@DinoChiesa I was actually just doing this yesterday, here’s the query I ended up to get the ratio of 4xx for some Istio gateways:
The big you want to add is something like:
This issue has actually been blocked internally since @pintohutch’s last comment, but we hope to be unblocked soon and can post updates here as we get more information
Hi @robmonct,
Yes. We’re tracking that work here and will hopefully have some updates soon.