opentelemetry-collector-contrib: Scraped Prometheus Metric Metadata Missing

Describe the bug We use the otel-collector to scrape metrics which have metadata about them. The best example of this is metrics that come from HAProxy’s Prometheus exporter. When the otel-collector ingests these and then sends them to Prometheus using the prometheusremotewrite exporter, the metadata doesn’t seem to make it. That said, the metadata does appear when using the logging exporter.

Steps to reproduce Scrape metrics with metadata associated with them and you will not see them. For example, the following is in the logs:

Descriptor:
     -> Name: haproxy_frontend_http_requests_total
     -> Description: Total number of HTTP requests processed by this object since the worker process started
     -> Unit:
     -> DataType: Sum
     -> IsMonotonic: true

You won’t see this in Grafana for example:

image

What did you expect to see?

I expected to see something similar to what is described in this Grafana article:

image

What did you see instead?

image

What version did you use?

aws-otel-collector:v0.20.0

What config did you use?

A trimmed down look at the basics of the config we are using:

  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 15s
          static_configs:
            - targets:
              - sometarget

exporters:
   # log level, defaults to info automatically on all outputs
  logging/metrics:
    loglevel: debug

  prometheusremotewrite:
    endpoint: <some_amazon_managed_prometheus_workspace>/api/v1/remote_write
    auth:
      authenticator: sigv4auth
    resource_to_telemetry_conversion:
      enabled: true

service:
  extensions:
    - sigv4auth
  pipelines:
    metrics:
      receivers:
        - prometheus
      exporters:
        - logging/metrics
        - prometheusremotewrite

Environment

Amazon Linux

Additional context Add any other context about the problem here.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 18 (18 by maintainers)

Commits related to this issue

Most upvoted comments

🤞 I’ll make some progress on this during the weekend.

@jmichalek132 nice! Shall I assign this issue to you then?

Yes please.

Seems like there is support for ingesting this at least with mimir, I would be interesting than working on implementing support for sending the metadata.