opentelemetry-collector-contrib: [processor/attributes] resource attributes sent to incorrect tenant

Describe the bug multi-tenant use cases in the Loki exporter were introduced through https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/12415, which allows setting and creating tenantIDs from resource attribute values. This is awesome and is the exact functionality we are looking for, but we are somehow seeing attribute values being sent to the wrong tenant, especially when logging at scale.

For example, if we configure the loki exporter to set the tenantID from the “namespace” attribute, we are seeing many logs from namespace foo show up in the namespace bar tenant in Loki.

Steps to reproduce This has been occurring using the opentelemetry collector helm chart with the following configuration. We are following the recommendations in the documentation to use the groupbyattrs processor, as well as batching.

config:
  receivers:
    fluentforward:
      endpoint: 0.0.0.0:8006
  processors:
    groupbyattrs:
      keys:
        - namespace_name
    filter/exclude:
      logs:
        exclude:
          match_type: regexp
          resource_attributes:
            - Key: namespace_name
              Value: istio-system|kube-system
  exporters:
    loki:
      endpoint: http://loki-distributed-gateway.loki.svc.cluster.local/loki/api/v1/push
      format: json
      tenant:
        source: attributes
        value: namespace_name
      labels:
        resource:
          namespace_name: ""
        attributes:
          stream: ""
          app: ""
          pod_name: ""
          account: ""
          cluster_name: ""
          region: ""
          container_name: ""
  service:
    pipelines:
      logs:
        exporters: [loki]
        processors: [memory_limiter, batch, groupbyattrs, filter/exclude]
        receivers: [fluentforward]

To reproduce, we noticed this occur more frequently during spiky loads at a higher volume, but also noticed it occur sometimes when the opentelemetry collector was killed or rescheduled onto a new node.

As part of troubleshooting we have also tested multiple combinations of this config, such as batching and grouping after the filter, as well as before, and also not batching at all. None of these have resulted in proper tenants.

What did you expect to see? We expect to see all logs from namespace foo only be in tenant foo and all logs from namespace bar only be in tenant bar.

What did you see instead? Logs from every namespace appear in other tenants.

below are some screenshots from Grafana where we expect to only see logs from the foo namespace.

Instead, we see many logs from other namespaces as well (even though our X-Scope-OrgID is set to the foo tenant)

What version did you use? Version: v0.57.2

Additional context We noticed in testing environments that with small non-spiky loads the logs were grouped and exported to the correct tenant, but in high-volume spiky environments the logs were not.