alloy: Various "error":"proto: wrong wireType = 2 for field TimeUnixNano" messages when using otel.receiver.kafka

What’s wrong?

Our architecture is currently:

Applications -> OTel Collector (otel/opentelemetry-collector-contrib:0.98.0) -> Kafka -> Grafana Alloy (1.0.0) -> Grafana Tempo (2.4.0).

OTel Collector receives the traces from our apps, and exports them onto a Kafka topic, using the otlp_proto encoding. We’ve then setup the otel.receiver.kafka component in Alloy to consume from this Kafka topic, to then send onto Tempo via gRPC (as we’re looking to enable Tempo Multi-tenancy, which isn’t supported by Kafka).

When Alloy starts consuming these messages, our Alloy logs start throwing various errors around wrong wireTypes, illegal wireTypes etc, which make no sense (previously when Kafka was being consumed by Tempo directly, we saw none of these errors).

Steps to reproduce

  1. Run a simple pipeline in OTel Collector on 0.98.0, exporting to a Kafka topic using the otlp_proto encoding format.
  2. Run a simple Alloy pipeline to consume from Kafka, batch process then output via gRPC to Tempo
  3. Watch the errors come in.

System information

No response

Software version

No response

Configuration

ALLOY:

otelcol.receiver.kafka "otelkafka" {
  brokers          = ["kafka-cluster-kafka-bootstrap.kafka.svc.cluster.local:9093"]
  protocol_version = "2.8.0"
  topic = "otlp-tracing"
  group_id = "alloy-otelkafka"
  client_id = "alloy-otelkafka"
  encoding = "otlp_proto"

  authentication {
    tls {
        ca_file = "/tmp/ca.crt"
        insecure = true
    }
  }

  message_marking {
    after_execution = true
    include_unsuccessful = true
  }

  output {
    metrics = [otelcol.processor.batch.otelkafka.input]
    logs    = [otelcol.processor.batch.otelkafka.input]
    traces  = [otelcol.processor.batch.otelkafka.input]
  }
}

otelcol.processor.batch "otelkafka" {
    send_batch_size = 10
    timeout = "100ms"
    output {
        metrics = [otelcol.exporter.otlp.otelkafka.input]
        logs    = [otelcol.exporter.otlp.otelkafka.input]
        traces  = [otelcol.exporter.otlp.otelkafka.input]
    }
}

otelcol.exporter.otlp "otelkafka" {
  client {
    endpoint = "grafana-tempo-distributed-distributor-discovery.monitoring.svc.cluster.local:4317"
    tls {
        insecure = true
        insecure_skip_verify = true
    }
  }
}

Logs

{"ts":"2024-04-23T09:17:23.578960444Z","level":"error","msg":"failed to unmarshal message","component_path":"/","component_id":"otelcol.receiver.kafka.otelkafka","error":"proto: illegal wireType 7"}
{"ts":"2024-04-23T09:17:23.582466437Z","level":"error","msg":"failed to unmarshal message","component_path":"/","component_id":"otelcol.receiver.kafka.otelkafka","error":"proto: wrong wireType = 2 for field TimeUnixNano"}
{"ts":"2024-04-23T09:17:32.771035129Z","level":"error","msg":"failed to unmarshal message","component_path":"/","component_id":"otelcol.receiver.kafka.otelkafka","error":"proto: Gauge: wiretype end group for non-group"}
{"ts":"2024-04-23T09:17:32.775138857Z","level":"error","msg":"failed to unmarshal message","component_path":"/","component_id":"otelcol.receiver.kafka.otelkafka","error":"proto: wrong wireType = 2 for field TimeUnixNano"}

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 20 (20 by maintainers)

Most upvoted comments

Sounds good, I can draft something for the receivers

This reminds me of #251. I think we need to make the otelcol.receiver components smart enough to only start a metric/traces/logs receivers if they are actually configured for the given component. E.g. if the otelcol.receiver component isn’t configured to output metrics downstream, it shouldn’t start a metrics receiver.

Hey @wildum,

Really appreciate you looking into this and giving some detailed analysis.

We’ve created the new Kafka topic ‘otlp_spans’ and got this running in one of our Dev environments, and can confirm this is working with no errors from Alloy at all (and all traces are being received by Tempo).