swift-otel: Metric export error sending to New Relic OTLP endpoint
I’ve recently started using this library to export metrics from a Vapor API to our New Relic account over gRPC
It all seems to work correctly, with the exception that every export logs an error of ok (0)
This is the error that is sent to our logging backend
{
"logLevel":"error",
"message":"Failed to export metrics.",
"metadata":{
"_file":"OTelPeriodicExportingMetricsReader.swift",
"_function":"tick()",
"_line":"61",
"_source":"OTel",
"error":"ok (0)"
}
}
Given the data is processed correctly, and the error message is ok I think somehow this is falsely getting marked as an error? Could it perhaps be some kind of missing configuration on my end?
My setup code is:
// Configure OTel resource detection to automatically apply helpful attributes to events.
let environment = OTelEnvironment.detected()
let resourceDetection = OTelResourceDetection(detectors: [
OTelProcessResourceDetector(),
OTelEnvironmentResourceDetector(environment: environment),
.manual(
OTelResource(
attributes: [
"service.name": "service-name",
"service.instance.id": "\(taskID)"
]
)
),
])
let resource = await resourceDetection.resource(environment: environment, logLevel: .trace)
// Bootstrap the metrics backend to export metrics periodically in OTLP/gRPC.
let registry = OTelMetricRegistry()
let metricsExporter = OTLPGRPCMetricExporter(
configuration: try OTLPGRPCMetricExporterConfiguration(
environment: environment,
endpoint: "otlp.nr-data.net:4317",
headers: HPACKHeaders(httpHeaders: HTTPHeaders([("api-key", Environment.get("NEW_RELIC_LICENSE_KEY") ?? "")]))
)
)
let metrics = OTelPeriodicExportingMetricsReader(
resource: resource,
producer: registry,
exporter: metricsExporter,
configuration: .init(
environment: environment
)
)
MetricsSystem.bootstrap(OTLPMetricsFactory(registry: registry))
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 16 (7 by maintainers)
@slashmo wrote:
Makes sense—thanks for checking.
Is there anything in the response we can use to distinguish the upstream as a New Relic service?
@czechboy0 wrote:
That they have deliberately called this out in their documentation as a clarification, likely means they do not see this as a bug.
At the end of the day this is an interop library and presumably we want adopters to be able to work with implementations across the ecosystem. For quirky implementations, we need to provide some way for that to work. Whether we bake in an implementation-specific workaround or provide the API surface needed for adopters to install their own workaround is an open question.
@slashmo on re-reading the spec, I’m now less convinced it’s a direct contravention of the spec. The part you linked is actually a subsection of the OTLP/HTTP requirements:
The part I linked is from the OTLP/GRPC section:
IMHO, the OTLP/GRPC section is implying the success response should contain a message, but it’s not sprinkling the MUST directives as liberally as in the OLTP/HTTP section.
But, again, based on the fact that it’s called out in the New Relic docs, it seems this is a conscious decision on their part.
OK. If folks feel strongly one way or another about whether we should log this error and how, then can I suggest we take it to a different thread/issue.
We can use this one to work out why we had an error thrown whose contents appears to suggest things were all fine.
@slashmo let me know if you are able to repro with your setup. I’ll be able to look at this but not super quickly.
Unfortunately I’m not quite sure how to set up a reproducer given the need for API keys… But I was just looking at their OTLP troubleshooting page (https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp/#payloads) and saw this:
Which sounds like it might be related to the “false” error log?
As for the impact, its purely a nuisance as it pollutes the logs quite a bit