helidon: [2.x] Prometheus unable to parse Helidon MP Exemplars

Environment Details

Helidon Version: 2.5.6
Helidon SE or Helidon MP : MP
JDK version: 17
OS: Oracle Linux 9
Docker version (if applicable):

Problem Description

The following is added to pom.xml to enable exemplars:

<dependency>
            <groupId>io.helidon.metrics</groupId>
            <artifactId>helidon-metrics-trace-exemplar</artifactId>
            <scope>runtime</scope>
</dependency>

Exemplars are then automatically added to metrics of type ‘gauge’ for JAX-RS annotated Services. This results in Prometheus ServiceMonitor failing to parse with the following error:

‘metric name base_REST_request_elapsedTime_seconds does not support exemplars’

snip from /metrics EP:

# TYPE base_REST_request_elapsedTime_seconds gauge
base_REST_request_elapsedTime_seconds{class="com.somepackage.HTMLService",method="getContent_java.lang.String_java.lang.String_java.lang.String_int_java.lang.String"} 49.343265107 # {trace_id="b17d175d1bfbcd8f"} 49343265107 1677598231.371000

Code snippet:

...
...
    @GET
    @Path("/content")
    public Response getContent(@QueryParam("secureURL") String secureURL,
                               @QueryParam("headerKey") String headerKey,
                               @QueryParam("headerVal") String headerVal,
                               @DefaultValue("300000") @QueryParam("timeout") int timeout,
                               @HeaderParam("x-request-id") String xRequestId) {
...
...
}

According to the following link, gauges are not supposed to be supported and suffix needs to be either _total or _bucket: https://github.com/prometheus/prometheus/issues/9738

Reading the specs, it is unclear if this should be the case. Would like to know a workaround/solution to this

About this issue

Original URL
State: closed
Created a year ago
Comments: 17 (9 by maintainers)

Most upvoted comments

Thanks very much for the test run.

Yes, we are targeting this fix for 2.6.0 which we plan to release very soon…hoping for early next week.

tjquinno on Mar 8, 2023

Thanks for the added information.

Helidon should not by itself be adding exemplars for metrics that the OpenMetrics spec does not say supports exemplars. https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md

I’m in the process of working on a fix.

That said, I still think it’s overly intolerant of the Prometheus service monitor to reject output that contains unexpected exemplars. After all, exemplars are expressed as comments in the output.

I might suggest you also file an issue against the service monitor. I would absolutely get it if the service monitor logged warnings about such unexpected exemplars and then consumed the data anyway (ignoring, of course, the unexpected exemplar data itself).

But to outright reject output based on what’s in comments does not seem right.

tjquinno on Mar 1, 2023