opentelemetry-collector-contrib: [resourcetotelemetry] - Using resource_to_telemetry_conversion enabled causes broken internal otelcol_ metrics

What happened?

Description

There is a prometheus exporter error failed to convert metric otelcol_xxxxxxxxxxxxx: duplicate label names if you enable

resource_to_telemetry_conversion:
      enabled: true

Steps to Reproduce

Add a receiver :

prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs:
            - targets: ['localhost:8888']

and enable resource_to_telemetry_conversion at prometheus exporter level. (I didn’t tested yet with prometheusremotewriteexporter)

Collector version

0.61.0

Environment information

Environment

OS: windows server 2019

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs:
            - targets: ['localhost:8888']
        - job_name: windows-exporter
          scrape_interval: 10s
          static_configs:
            - targets: ['localhost:9182']

exporters:
  prometheus:
    endpoint: "0.0.0.0:9095"
    send_timestamps: true
    metric_expiration: 30s
    enable_open_metrics: false
    resource_to_telemetry_conversion:
      enabled: true

processors:
  batch:
    send_batch_size: 50
    timeout: 5s
  resourcedetection:
    detectors: ["gke","aks","eks","ec2","gce","azure","ecs"]
    ec2:
      tags:
        - ^Name$
  resource/metrics:
    attributes:
    - key: job_from
      from_attribute: job
      action: insert
    #- key: job
    #  action: delete
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_percentage: 30

extensions:
  health_check:
  memory_ballast:
    size_mib: 32

service:
  extensions: [memory_ballast,health_check]
  telemetry:
    logs:
      level: info
    metrics:
      level: normal
      address: ":8888"
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, resourcedetection, resource/metrics, batch]
      exporters: [prometheus]

Log output

2022-10-12T12:11:59.062Z        error   prometheusexporter@v0.61.0/collector.go:367     failed to convert metric otelcol_process_cpu_seconds: duplicate label names     {"kind": "exporter", "data_type": "metrics", "name": "prometheus"
}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.61.0/collector.go:367
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.13.0/prometheus/registry.go:455
2022-10-12T12:11:59.067Z        error   prometheusexporter@v0.61.0/collector.go:367     failed to convert metric otelcol_processor_batch_batch_send_size: duplicate label names {"kind": "exporter", "data_type": "metrics", "name": "pro
metheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.61.0/collector.go:367
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.13.0/prometheus/registry.go:455
2022-10-12T12:11:59.067Z        error   prometheusexporter@v0.61.0/collector.go:367     failed to convert metric otelcol_exporter_enqueue_failed_spans: duplicate label names   {"kind": "exporter", "data_type": "metrics", "name": "pro
metheus"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).Collect
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.61.0/collector.go:367
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        github.com/prometheus/client_golang@v1.13.0/prometheus/registry.go:455

Additional context

No response

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 4
Comments: 16 (12 by maintainers)

Most upvoted comments

I have fixed the issue using the metric relabeling configuration in the receiver:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector-self'
          scrape_interval: 30s
          static_configs:
            - targets: ['localhost:8888']
          metric_relabel_configs:
            - action: labeldrop
              regex: "service_instance_id|service_name"

Explanation

Prometheus exporter is automatically generating not only service.instance.id label (thanks @gillg), but in my case it was also rendering new service.name label value. Original service.name label value on the receiver was otelcol-contrib, however, the exporter is setting that value to match job name, in this case otel-collector-self.

Collector version I have tested this on is 0.78.

krsmanovic on Aug 3, 2023

I found the reason, the duplicated label is service.instance.id. Internaly the value is a random uuid, but once scrapped by prometheus receiver, the “instance” become “localhost:8888”. Then if we enable resources as attributes, the instance is converted to service.instance.id and it cause a duplicated label.

gillg on Oct 12, 2022