opentelemetry-collector-contrib: Scraped Prometheus Metric Metadata Missing
Describe the bug
We use the otel-collector to scrape metrics which have metadata about them. The best example of this is metrics that come from HAProxy’s Prometheus exporter. When the otel-collector ingests these and then sends them to Prometheus using the prometheusremotewrite
exporter, the metadata doesn’t seem to make it. That said, the metadata does appear when using the logging exporter.
Steps to reproduce Scrape metrics with metadata associated with them and you will not see them. For example, the following is in the logs:
Descriptor:
-> Name: haproxy_frontend_http_requests_total
-> Description: Total number of HTTP requests processed by this object since the worker process started
-> Unit:
-> DataType: Sum
-> IsMonotonic: true
You won’t see this in Grafana for example:
What did you expect to see?
I expected to see something similar to what is described in this Grafana article:
What did you see instead?
What version did you use?
aws-otel-collector:v0.20.0
What config did you use?
A trimmed down look at the basics of the config we are using:
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 15s
static_configs:
- targets:
- sometarget
exporters:
# log level, defaults to info automatically on all outputs
logging/metrics:
loglevel: debug
prometheusremotewrite:
endpoint: <some_amazon_managed_prometheus_workspace>/api/v1/remote_write
auth:
authenticator: sigv4auth
resource_to_telemetry_conversion:
enabled: true
service:
extensions:
- sigv4auth
pipelines:
metrics:
receivers:
- prometheus
exporters:
- logging/metrics
- prometheusremotewrite
Environment
Amazon Linux
Additional context Add any other context about the problem here.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 18 (18 by maintainers)
Commits related to this issue
- [exporter/prometheusremotewrite] prometheusremotewrite exporter add option to send metadata (#27565) **Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how thi... — committed to open-telemetry/opentelemetry-collector-contrib by jmichalek132 7 months ago
- [exporter/prometheusremotewrite] prometheusremotewrite exporter add option to send metadata (#27565) **Description:** <Describe what has changed.> <!--Ex. Fixing a bug - Describe the bug and how thi... — committed to ClickHouse/opentelemetry-collector-contrib by jmichalek132 7 months ago
🤞 I’ll make some progress on this during the weekend.
Yes please.
Seems like there is support for ingesting this at least with mimir, I would be interesting than working on implementing support for sending the metadata.