quarkus: traceId, spanId not set in logs when using quartz and opentelemetry

quarkus-opentelemetry

Gonna have a look at this

It would require adding proper OTel integration in a JobListener. This can be used as a basis: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/quartz-2.0/library/src/main/java/io/opentelemetry/instrumentation/quartz/v2_0

Would you like to try submitting a PR with the proper support? 😃

If everything works with @WithSpan, which we treat as an interceptor binding, enabling tracing by default for scheduled methods would amount to a simple annotation transformation I’d say.

@syr I’m not using the otel agent explicitly, only quarkus-opentelemetry + quarkus-scheduler dependencies and additionally io.opentelemetry.instrumentation:opentelemetry-jdbc.

@WithSpan just works, I see JDBC sub-spans for my job executions.

I created a separate enhancement issue: #29647

Wow this sounds really promissing and gives a very specific picture. Thanks for sharing @brunobat

I would wrap it up here now and close the ticket as we have a perspective for 2023.

For everyone looking for a workaround/fix to get traceId, spanId set in logs when using libs which are not instrumented by quarkus-opentelemetry atm (e.g. quartz, aws sdk) I woud advice to use otel agent, which requires:

1. updating the MDCs in code (required because of: https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/7120#issuecomment-1316782316):

        SpanContext spanContext = Span.current().getSpanContext();
        MDC.put("traceId", spanContext.getTraceId());
        MDC.put("spanId", spanContext.getSpanId());

1. providing a ThreadContextProvider to get MDCs propagated to logger in mutiny async threads, see: https://stackoverflow.com/questions/65039035/manual-context-propagation-in-quarkus-native-mode/65040601#65040601
2. To add the quarkus-opentelemetry dependency but disabling it in properties quarkus.opentelemetry.enabled=false to get the otel interface for example to create custom spans (e.g. @WithSpan)

OpenTelemetry tracing is not experimental anymore. We are preparing a few changes, soon, that will wide the instrumentation support.

OK sounds good. I was referring to the status shown at code.quarkus.io. Is there a rough roadmap weeks/months/2023 when to expect these extended otel support beeing released? This info would be very helpful to decide how much effort we put in workarounds using otel agent.

The OTel agent instrumentation is too generic. It seems to work but, as I mentioned, it has context propagation issues within the service, specially if you mix imperative and reactive code. The built in Quarkus instrumentation fixes that.

Yes I noticed quite some issues (mainly three points in https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/7120#issuecomment-1316697698), however it was the only way I see to get quartz and aws sdk auto-instrumented atm.

We have quite few reactive code in the project, I would rather consider to change it back to impierative because there are also issues when Mutiny is combined with @Transactional for example…likely mixing imperative and reactive is not a good idea. But that’s another topic…

Tracing does make sense on Quartz Jobs. This is not just for HTTP. It will be great to understand the outcome of each job. This requires 2 changes, though. One thing is to instrument jobs, another one is curating the MDC context with the right data.

Technically, a scheduled method can call other external services etc., and so tracing may make sense. As you mention, a scheduled method is an entrypoint, so it deserves a fresh new trace/span, I’d say.