opentelemetry-collector: [processor/attributes] unable to add span custom attribute from custom header

Describe the bug By enabling include_metadata and using attributes processor I don’t see http headers from the POST request on the otlp http endpoint added as span attributes.

Steps to reproduce I have set include_metadata to true in my http otlp receiver I try to upsert a span custom attribute using attributes processor just like in this readme:

https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md

In my case the header is CloudFront-Viewer-Longitude but I don’t think that’s relevant :

          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert

Anyway below you’ll find my full configuration.

What did you expect to see? CloudFront-Viewer-Longitude added as http.client_longitude span attribute.

What did you see instead? http.client_longitude is not added among span attributes.

What version did you use? otel/opentelemetry-collector-contrib:0.54.0

What config did you use?

receivers:
    otlp:
        protocols:
            http:
                include_metadata: true
                cors:
                    allowed_origins: [
                        ${RECEIVERS_OTLP_PROTOCOLS_HTTP_CORS_ALLOWED_ORIGINS}
                    ]
                    
    # Dummy receiver that's never used, because a pipeline is required to have one.
    otlp/spanmetrics:
        protocols:
            grpc:
                endpoint: "localhost:12345"

    prometheus/spanmetrics:
      config:
        scrape_configs:
          - job_name: 'ratelimiter'
            scrape_interval: 6m
            static_configs:
              - targets: ['localhost:8889']

processors:
    attributes/traces:
        actions:
          - key: http.client_latitude_2
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude_2
            from_context: metadata.CloudFront-Viewer-Longitude
            action: upsert
          - key: http.client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: http.client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert
          - key: test
            value: 123
            action: upsert
          - key: origin2
            from_context: metadata.origin
            action: upsert
          - key: client_latitude
            from_context: CloudFront-Viewer-Latitude
            action: upsert
          - key: client_longitude
            from_context: CloudFront-Viewer-Longitude
            action: upsert
          - key: client_latitude2
            from_context: cloudfront-viewer-latitude
            action: upsert
          - key: client_longitude2
            from_context: cloudfront-viewer-longitude
            action: upsert
          - key: client_latitude3
            from_context: metadata.cloudfront-viewer-latitude
            action: upsert
          - key: client_longitude3
            from_context: metadata.cloudfront-viewer-longitude
            action: upsert
          - key: client_latitude4
            from_context: metadata.CloudFront-Viewer-Latitude
            action: upsert
          - key: client_longitude4
            from_context: metadata.CloudFront-Viewer-Longitude
            action: upsert
    memory_limiter:
        check_interval: 5s
        limit_mib: 448
        spike_limit_mib: 64
    batch:
        send_batch_size: 48
        send_batch_max_size: 48
        timeout: 15s
    transform:
        logs:
            queries:
                - set(attributes["severity_text"], severity_text)
    filter:
        logs:
            exclude:
                match_type: regexp
                record_attributes:
                    - key: severity_text
                      value: "(TRACE|DEBUG)"
    spanmetrics:
        metrics_exporter: prometheus/spanmetrics
        dimensions:
            - name: enduser.id
            - name: http.status_code
            - name: http.client_ip
            - name: http.client_latitude
            - name: http.client_longitude
            - name: http.method
        aggregation_temporality: "AGGREGATION_TEMPORALITY_DELTA"

exporters:
    logging:
        loglevel: ${EXPORTERS_LOGGING_LOGLEVEL}
    otlp:
        endpoint: tempo-eu-west-0.grafana.net:443
        headers:
            authorization: ##########
            
    prometheusremotewrite:
        endpoint: https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push
        headers:
            authorization: ##########
            
    loki:
        endpoint: https://logs-prod-eu-west-0.grafana.net/loki/api/v1/push
        headers:
            authorization: ##########
        format: json
        labels:
            attributes:
                container_name: ""
                source: ""
            resource:
                host.name: "hostname"
                
    prometheus/spanmetrics:
        endpoint: "localhost:8889"

extensions:
    health_check:
service:
    telemetry:
        logs:
            level: ${SERVICE_TELEMETRY_LOGS_LEVEL}
    extensions: [health_check]
    pipelines:
        traces:
            receivers: [otlp]
            processors: [attributes/traces, spanmetrics, memory_limiter, batch]
            exporters: [otlp, logging]
        metrics/spanmetrics:
            # This receiver is just a dummy and never used.
            # Added to pass validation requiring at least one receiver in a pipeline.
            receivers: [otlp/spanmetrics]
            # The metrics_exporter must be present in this list.
            exporters: [prometheus/spanmetrics]
        metrics:
            receivers: [otlp, prometheus/spanmetrics]
            processors: [memory_limiter, batch]
            exporters: [prometheusremotewrite]
        logs:
            receivers: [otlp]
            processors: [transform, filter, memory_limiter, batch]
            exporters: [loki]

Environment AWS Beanstalk with Docker Linux V2

Additional context

I need this to inject AWS Cloudfront geolocation headers into my spans : https://aws.amazon.com/it/about-aws/whats-new/2020/07/cloudfront-geolocation-headers/

I have put my otel collector behind a non caching Cloudfront, and it is properly receiving POST http requests on v1/traces with those headers containing latitude and logitude.

I am sure everything is received properly because I am running tcpdump on 4318 port on otel collector and I DO see those headers.

Now I need to propagate those values from the http request, containing the spans in its body, to the span themself.

This is what include_metadata is supposed to enable, right ? Did I understand correctly ?

I also tried different combinations, trying to read from metadata context (inspired by https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/attributesprocessor/testdata/config.yaml) but none worked.

The only attribute I see properly set is “test” to value 123, but that’s the only key not using “from_context”.

Please help.