prometheus-engine: Managed Prometheus does not support _count exemplars

According to this page: https://cloud.google.com/stackdriver/docs/managed-prometheus/exemplars There is no support for counter exemplars: “Exemplars attached to counter metrics can’t be ingested”

The problem is that we are using spring-boot, and they have added exemplars to other types: https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.2-Release-Notes#broader-exemplar-support-in-micrometer-112

as this is available for all supported prometheus version.

This leads to us not being able to scrape any metrics when using managed prometheus. Port-forwarding to collectors pods, and checking the GUI, shows errors like this:

metric name http_server_requests_seconds_count does not support exemplars

Is there any timeline for when managed prometheus will update to prometheus v 2.43 or later, and start supporting exemplars for other types than histograms?

This is currently preventing us from fully migrating away from our own prometheus to the managed one.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 16 (4 by maintainers)

Most upvoted comments

Is there any timeline for when managed prometheus will update to prometheus v 2.43 or later

To answer this part of the question, we are currently deploying Prometheus v2.43 out to the GKE Rapid release channel for new clusters within the next few days.

We believe the upgrade should suppress the error but the exemplars will be silently dropped, although we still need to test and verify this.

It’s pretty unlikely we will support counter exemplars in the near term. In Monarch, exemplars are stored with the point itself, and only histograms have the necessary data structure (proto) for that. Counters and Gauges are stored far more efficiently and there’s no place to put an exemplar. Changing this would likely require us building an entirely separate exemplar storage system, which isn’t likely given that most of the need for exemplars is on histograms.

That being said, we should probably give you a way to ingest these metrics without exemplars (dropping the exemplar + ingesting the time series, instead of erroring) so that you don’t have to modify the exporter to get these metrics. I’ll open a FR for this, thank you for letting us know!

+1 - agreed on your assessment, upon further review it looks like once 2.45 is rolled out this issue will fix itself automatically. We’re still qualifying it… should be soon though.