opentelemetry-java-instrumentation: Cannot get Micrometer custom metrics

Describe the bug I have a Java application instrumented to create custom Micrometer metrics. The very same Java application uses the OpenTelemetry Java Instrumentation agent to collect metrics using the Prometheus exporter.

I can see Micrometer metrics and OpenTelemetry metrics being exported just fine, but I cannot see Micrometer metrics exported via OpenTelemetry Prometheus exporter.

It’s my understanding that from 1.10 the agent should export Micrometer metrics but maybe I get this wrong.

Agent is configured like this:

-Dotel.service.name=webportal
-Dotel.exporter.otlp.endpoint=http://tempo.tempo.svc.cluster.local:4317
-Dotel.traces.exporter=otlp
-Dotel.exporter.otlp.compression=gzip
-Dotel.metrics.exporter=prometheus

Querying port 9464 produces OTEL metrics like the following:

http_client_duration_sum{http_flavor="1.1",http_method="GET",http_status_code="200",http_url="http://localhost:8080/webtng/ediMonitorREST_1_0_0/status",} 301.605403 1643730489470
http_client_duration_bucket{http_flavor="1.1",http_method="GET",http_status_code="200",http_url="http://localhost:8080/webtng/ediMonitorREST_1_0_0/status",le="5.0",} 0.0 1643730489470
http_client_duration_bucket{http_flavor="1.1",http_method="GET",http_status_code="200",http_url="http://localhost:8080/webtng/ediMonitorREST_1_0_0/status",le="10.0",} 0.0 1643730489470
http_client_duration_bucket{http_flavor="1.1",http_method="GET",http_status_code="200",http_url="http://localhost:8080/webtng/ediMonitorREST_1_0_0/status",le="25.0",} 0.0 1643730489470

Querying the Mircometer port I get the following:

edimonitor_inmessage_incorrect_total{application="edimonitor",edimonitorinstance="0",edimonitorjob="JOB",} 0.0

but I cannot find the latter metrics inside the former ones.

Steps to reproduce

instrument a Java application with custom Java Micrometer metrics
auto-instrument a Java application with OTEL agent
enable Prometheus exporter for OTEL
verify that Micrometer metrics are exported via OTEL agent

What did you expect to see? Metrics from Micrometer reported in the Prometheus exporter endpoint

What did you see instead? Metrics from Micrometer are not reported in the Prometheus exporter endpoint

What version are you using? 1.10.1

Environment Compiler: (e.g., “AdoptOpenJDK 11.0.6”) OS: (e.g., “Ubuntu 20.04”) Runtime (if different from JDK above): (e.g., “Oracle JRE 8u251”) OS (if different from OS compiled on): (e.g., “Windows Server 2019”)

Additional context Add any other context about the problem here.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 20 (9 by maintainers)

Most upvoted comments

Ok @mateuszrzeszutek, after testing the solution it looks like:

PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
        prometheusRegistry.config()
                .meterFilter(new PrometheusRenameFilter())
                .pauseDetector(new ClockDriftPauseDetector(Duration.of(100, ChronoUnit.MILLIS), Duration.of(100, ChronoUnit.MILLIS)));
Metrics.globalRegistry.add(prometheusRegistry);

plus registering the custom metrics to the global registry only, allowed me to get the custom metrics from both OTEL Prometheus exporter and the Micrometer endpoint.

Thank you very much for your help

irizzant on Feb 1, 2022

Thanks @irizzant. your example resolved my doubts & I can send my custom metrics to open telemetry. I have directly sent my custom metrics to global registry & it showed up in the open telemetry metrics. Example code

Timer.builder("Test Metrics") .tags(Tags.of(Tag.of("Tag -1", "A"), Tag.of("Tag-2", "B"))) .register(Metrics.globalRegistry)

patel-prakash on Apr 27, 2023