swift-otel: Metric export error sending to New Relic OTLP endpoint

I’ve recently started using this library to export metrics from a Vapor API to our New Relic account over gRPC

It all seems to work correctly, with the exception that every export logs an error of ok (0)

This is the error that is sent to our logging backend

{
     "logLevel":"error",
     "message":"Failed to export metrics.",
     "metadata":{
          "_file":"OTelPeriodicExportingMetricsReader.swift",
          "_function":"tick()",
          "_line":"61",
          "_source":"OTel",
          "error":"ok (0)"
     }
}

Given the data is processed correctly, and the error message is ok I think somehow this is falsely getting marked as an error? Could it perhaps be some kind of missing configuration on my end?

My setup code is:

        // Configure OTel resource detection to automatically apply helpful attributes to events.
        let environment = OTelEnvironment.detected()
        let resourceDetection = OTelResourceDetection(detectors: [
            OTelProcessResourceDetector(),
            OTelEnvironmentResourceDetector(environment: environment),
            .manual(
                OTelResource(
                    attributes: [
                        "service.name": "service-name",
                        "service.instance.id": "\(taskID)"
                    ]
                )
            ),
        ])
        let resource = await resourceDetection.resource(environment: environment, logLevel: .trace)
        
        // Bootstrap the metrics backend to export metrics periodically in OTLP/gRPC.
        let registry = OTelMetricRegistry()
        let metricsExporter = OTLPGRPCMetricExporter(
            configuration: try OTLPGRPCMetricExporterConfiguration(
                environment: environment,
                endpoint: "otlp.nr-data.net:4317",
                headers: HPACKHeaders(httpHeaders: HTTPHeaders([("api-key", Environment.get("NEW_RELIC_LICENSE_KEY") ?? "")]))
            )
        )
        let metrics = OTelPeriodicExportingMetricsReader(
            resource: resource,
            producer: registry,
            exporter: metricsExporter,
            configuration: .init(
                environment: environment
            )
        )
        MetricsSystem.bootstrap(OTLPMetricsFactory(registry: registry))

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 16 (7 by maintainers)

Most upvoted comments

@slashmo wrote:

I was now able to reproduce this issue with my NewRelic account and the Swift OTel Server Example. As expected, the issue does not only occur for metrics exports but also for span exports.

Makes sense—thanks for checking.

I was able to mitigate it with the following interceptor (only checked Metrics but Tracing should work the same)…

Is there anything in the response we can use to distinguish the upstream as a New Relic service?


@czechboy0 wrote:

If we double check that New Relic is really not compliant with OTLP, should we be adding any workarounds here? Can we report it to them? Have other otel implementations added a similar workaround for New Relic?

That they have deliberately called this out in their documentation as a clarification, likely means they do not see this as a bug.

At the end of the day this is an interop library and presumably we want adopters to be able to work with implementations across the ecosystem. For quirky implementations, we need to provide some way for that to work. Whether we bake in an implementation-specific workaround or provide the API surface needed for adopters to install their own workaround is an open question.


@slashmo on re-reading the spec, I’m now less convinced it’s a direct contravention of the spec. The part you linked is actually a subsection of the OTLP/HTTP requirements:

OTLP/HTTP Response

Response body MUST be the appropriate serialized Protobuf message (see below for the specific message to use in the Success and Failure cases).

The server MUST set “Content-Type: application/x-protobuf” header if the response body is binary-encoded Protobuf payload. The server MUST set “Content-Type: application/json” if the response is JSON-encoded Protobuf payload. The server MUST use the same “Content-Type” in the response as it received in the request.

If the request header “Accept-Encoding: gzip” is present in the request the server MAY gzip-encode the response and set “Content-Encoding: gzip” response header.

Success

On success the server MUST respond with HTTP 200 OK. Response body MUST be Protobuf-encoded ExportTraceServiceResponse message for traces, ExportMetricsServiceResponse message for metrics and ExportLogsServiceResponse message for logs.

The server SHOULD respond with success no sooner than after successfully decoding and validating the request. — source: https://github.com/open-telemetry/opentelemetry-specification/blob/v1.8.0/specification/protocol/otlp.md#success

The part I linked is from the OTLP/GRPC section:

The server may respond with either a success or an error to the requests.

The success response indicates telemetry data is successfully processed by the server. If the server receives an empty request (a request that does not carry any telemetry data) the server SHOULD respond with success.

Success response is returned via Export*ServiceResponse message (ExportTraceServiceResponse for traces, ExportMetricsServiceResponse for metrics, ExportLogsServiceResponse for logs). — source: https://github.com/open-telemetry/opentelemetry-specification/blob/v1.8.0/specification/protocol/otlp.md#otlpgrpc-response

IMHO, the OTLP/GRPC section is implying the success response should contain a message, but it’s not sprinkling the MUST directives as liberally as in the OLTP/HTTP section.

But, again, based on the fact that it’s called out in the New Relic docs, it seems this is a conscious decision on their part.

OK. If folks feel strongly one way or another about whether we should log this error and how, then can I suggest we take it to a different thread/issue.

We can use this one to work out why we had an error thrown whose contents appears to suggest things were all fine.

@slashmo let me know if you are able to repro with your setup. I’ll be able to look at this but not super quickly.

Unfortunately I’m not quite sure how to set up a reproducer given the need for API keys… But I was just looking at their OTLP troubleshooting page (https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp/#payloads) and saw this:

Successful responses from New Relic have no response body, instead of a [Protobuf-encoded response](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.8.0/specification/protocol/otlp.md#success) based on the data type.

Which sounds like it might be related to the “false” error log?

As for the impact, its purely a nuisance as it pollutes the logs quite a bit