prometheus-engine: PromQL for Google Cloud Monitoring metrics doesn't support label regexs for integers

In Prometheus, all metric labels are strings, and PromQL supports label selection with regular expressions. In Cloud Monitoring, some metric labels are integers.

For example, we use a log based metric that counts HTTP requests by status code. The status code is INT64 label for the metric. Trying to query this metric using PromQL, we get the following error:

Error executing query: invalid parameter "query": substitute queries failed: convert vector selector failed: unable to parse value in matcher on int label to an int: strconv.ParseInt: parsing "5\\d\\d": invalid syntax.

The query is trying to count up the 5xx status codes, for use in an alert. It appears as if the label is an integer type, we can no longer use the label regex in PromQL.

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 17
  • Comments: 30 (2 by maintainers)

Most upvoted comments

PromQL replatforming is code complete, it will roll out this month. This is the first feature that we’ll deliver once that’s done. Thanks for keeping the pressure on 😃

Yeah - we’re super close to finishing the replatforming of PromQL, should be ready this month. At that point we’ll switch everyone over and this will work. 😃

OR, the other option is to allow filters that use comparison operators (greater-than, lesser-than, greater-than-or-equal, lesser-than-or-equal) against integers. So for example

rate(
      response_count{response_code>=400,response_code<500}[${__interval}]
    ),

I know this issue was originally about regex, but, the real problem (it seems to me) is some way to handle int labels with greater facility. Casting int to string implicitly and applying regex is one way to solve that problem. Comparison operators is another (potential) way to solve the problem . For me, anyway.

Adding >/< operators to PromQL would break spec with PromQL (as all label values in PromQL must be strings) and we really don’t want to do that. The real problem is that we even have INT value labels in the first place for GCP metrics. IMO the real solution is for us to be able to treat INT values like Strings, and then you can do whatever with PromQL and we don’t break compatibility.

Thanks for nudging me - we’re still working on this but unfortunately we’re not hitting the September date 😕

ETA early 2025, hopefully. It’s a top priority of ours.

Just a heads up that while using label_replace() might work, using it inside an aggregation function is likely to move that part of the query execution out of the very efficient Monarch query engine and into the much less efficient PromQL query engine. In other words, while what you’re doing might work just fine for small amounts of data, it is likely to time out or cause other errors for large amounts of data.

Nothing yet but this work is prioritized as part of a larger promql refactoring happening soon. I’d expect it to be unblocked sometime around September.

Code is done, but we’re in a big freeze because of the large GCP-wide outage a few weeks back. It’ll be rolled out as soon as that’s lifted.

@DinoChiesa I was actually just doing this yesterday, here’s the query I ended up to get the ratio of 4xx for some Istio gateways:

sum by (source_canonical_service_name,response_code_class)
(
  label_replace(
    rate(istio_io:service_server_request_count{monitored_resource="k8s_container",source_canonical_service_name=~"${gateways}"}[$__rate_interval]) >0,
    "response_code_class",
    "$1",
    "response_code",
    "(.).."
  ) and on (response_code_class) (
    label_replace(
      vector(1),
      "response_code_class",
      "4",
      "",
      ""
    )
  )
)
/ on (source_canonical_service_name)
sum by (source_canonical_service_name)
(
  rate(istio_io:service_server_request_count{monitored_resource="k8s_container",source_canonical_service_name=~"${gateways}"}[5m]) >0
)

The big you want to add is something like:

and on (status_group) (
    label_replace(
      vector(1),
      "status_group",
      "4xx",
      "",
      ""
    )
  )

This issue has actually been blocked internally since @pintohutch’s last comment, but we hope to be unblocked soon and can post updates here as we get more information

Hi @robmonct,

Yes. We’re tracking that work here and will hopefully have some updates soon.