opentelemetry-collector-contrib: Loki to Loki exporter not working

Component(s)

exporter/loki

What happened?

Description

I am trying to send data from promtail to loki via the OTEL collector. It’s does not appear to be sending to LOKI

Steps to Reproduce

Configure a loki receiver and loki exporter Send a message based on the example

curl -v -H "Content-Type: application/json" -XPOST -s "http://otelcol:3500/loki/api/v1/push" --data-raw   '{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "'$(date +'%s')'000000000", "fizzbuzz3" ] ] }]}'

Expected Result

Debug message appears AND data appears in loki

Actual Result

Debug message appears but no data appears in loki.

Sending via

curl -v -H "Content-Type: application/json" -XPOST -s "http://loki:3100/loki/api/v1/push" --data-raw   '{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "'$(date +'%s')'000000000", "fizzbuzz2" ] ] }]}'

Works and I receive the entry.

Collector version

v0.88.0

Environment information

Environment

OS: DietPi Docker Swarm

OpenTelemetry Collector configuration

exporters:
  debug:
    verbosity: detailed
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    default_labels_enabled:
      exporter: true
      job: true
receivers:
  loki:
    protocols:
      http:
    use_incoming_timestamp: true
processors:
  batch:
  attributes/loki:
    actions:
      - action: insert
        key: loki.attribute.labels
        value: container
      - action: insert
        key: loki.format
        value: raw
service:
  pipelines:
    logs:
      receivers: [loki]
       # I tried without batch even attributes
      processors: [attributes/loki,batch]
      # I also tried without debug
      exporters: [debug,loki]

Log output

{"level":"info","ts":1699392758.861195,"msg":"LogsExporter","kind":"exporter","data_type":"logs","name":"debug","resource logs":1,"log records":1}
{"level":"info","ts":1699392758.8614204,"msg":"ResourceLog #0\nResource SchemaURL: \nScopeLogs #0\nScopeLogs SchemaURL: \nInstrumentationScope  \nLogRecord #0\nObservedTimestamp: 2023-11-07 21:32:38.731536162 +0000 UTC\nTimestamp: 2023-11-07 21:32:38 +0000 UTC\nSeverityText: \nSeverityNumber: Unspecified(0)\nBody: Str(fizzbuzz3)\nAttributes:\n     -> foo: Str(bar2)\n     -> loki.attribute.labels: Str(container)\nTrace ID: \nSpan ID: \nFlags: 0\n","kind":"exporter","data_type":"logs","name":"debug"}

Additional context

No response

About this issue

Original URL
State: open
Created 8 months ago
Comments: 20 (11 by maintainers)

Most upvoted comments

right, all labels you configured in promtail will be lost in the collector unless you explicitly promote them to loki labels.

Thank you for the feedback regarding docs. Agree docs should make it clear that collector is not a proxy

mar4uk on Nov 8, 2023

Nope all it showed was this which was part of the tracing https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/29015

{
  "level": "debug",
  "ts": 1699396862.3295133,
  "caller": "ottl@v0.88.0/compare.go:22",
  "msg": "string to non-string",
  "kind": "processor",
  "name": "filter/drop_actuator",
  "pipeline": "traces",
  "op": 0
}

I turned off the trace processors to silence that log but all I see is the info log for the message I send

trajano on Nov 7, 2023

Checking, but I did copy that block from https://grafana.com/docs/opentelemetry/collector/send-logs-to-loki/loki-receiver/#use-opentelemetry-collector-to-send-logs-to-loki

Oh, thanks for linking, you’re right. We’ll have to check with the loki code owners to see if one is wrong (maybe they both work?), and update documentation if necessary 👍

crobert-1 on Nov 7, 2023