moleculer: Moleculer Datadog integration is not working

@icebob You mentioned several operations that are traced above. If you can link me to the code of one of them and also to how it’s currently traced I can try to do it using only our public interface if at all possible. If not, it will at least give me a better idea of how tracing works in Moleculer and what could be done to improve the interoperability.

@rochdev Yeah, this is currently a problem. The moleculer spans & datadog spans are not connected. The problem is that Moleculer is a full complex framework with a built-in tracing solution which monitors all services (actions, events, custom spans…etc). So you can’t create a dd-agent instrument for this, similar for express or redis where sufficient wrap some methods and the tracing will work.

In previous versions (maybe dd-trace 0.16) I’ve tried to make the connection between moleculer & datadog spans and it worked. I checked the full dd-trace source code and found the way how could I get the active span and how could I overwrite it from Moleculer (it was a dirty way because it uses private variables in dd-trace). But in the later dd-trace versions this internal things has been changed and this code stopped working. Here is a full trace from those times: You can see that the first span comes from DD-trace (http instrument) and there are Redis dd-trace spans between Moleculer spans.

In brief, I can solve it, if dd-trace would be two public methods. One method to get the current datadog span (trace_id & span_id & parent_span_id) based on the current executionAsyncId, and another method to set the current span (span_id & parent_span_id). Is there any chance that you can create them inside dd-trace?

I’m seeing this issue too. I filed https://github.com/DataDog/dd-trace-js/issues/970, but after more investigation I believe the core problem is moleculer improperly setting IDs on the dd-trace spans. dd-trace’s MessagePack encoder expects the traceId and spanId to be 8 bytes, because that’s what the id generator defaults to. However, moleculer overrides the dd-trace span’s generated IDs with the 16-byte GUIDs used by moleculer’s internal tracer. The extra bytes aren’t handled in the MessagePack encoding and it ends up corrupting the trace payload, resulting in decoding errors on the agent.

In addition to corrupting the trace payload, it seems to also corrupt some internal state in dd-trace because future trace payloads will also be corrupt, even if their ids are the expected 8 bytes. I haven’t yet gone far enough to figure out why this happens.

(Related: #690. Perhaps these issues can be solved at the same time.)

Update:

I have tried this by doing activate in tracing middleware just before calling the actual handler and scope propagates fine. I am not sure yet how to do it for all of the exporters

datadog.js (exporter)

span.meta.datadog = {
    span: ddSpan,
    ddTracer: this.ddTracer,
    asyncId,
    oldSpan
};

tracer.js (middleware)

span.meta.datadog.ddTracer.scope().activate(span.meta.datadog.span, () => {
    // Call the handler
    return handler(ctx).then(res => {
        const tags = {
            fromCache: ctx.cachedResult
        };

        if (_.isFunction(opts.tags)) {
            const r = opts.tags.call(ctx.service, ctx, res);
            if (r)
                Object.assign(tags, r);

        } else if (_.isPlainObject(opts.tags)) {
            if (opts.tags.response === true)
                tags.response = res != null ? Object.assign({}, res) : res;
            else if (Array.isArray(opts.tags.response))
                tags.response = _.pick(res, opts.tags.response);
        }

        span.addTags(tags);
        ctx.finishSpan(span);

        //ctx.duration = span.duration;

        return res;
    }).catch(err => {
        span.setError(err);
        ctx.finishSpan(span);

        throw err;
    });
})

CC: @icebob

@rochdev Great! In Moleculer there are actions and events. Both of them traced. Both have a Context instance which contains data for tracing, as well (id, parentID, requestID).

The built-in tracing feature is created as a middleware which can wrap the internal methods and can add additional functions. Here is the source of the tracing middleware

It wraps the service action & event handlers and generates tracing span based on the Context. It calls the ctx.startSpan and ctx.finishSpan methods which are a shorthand for the Tracer startSpan & span.finish

The Tracer has exporters, it can be multiple. One of them the Datadog exporter which uses the dd-trace to convert the Moleculer traces & spans to Datadog traces & spans.

The developers can create custom spans as well. For this they call the ctx.startSpan()/ctx.finishSpan() methods. Or if Context is not available, broker.tracer.startSpan()/span.finish()

When a request goes from NodeA to NodeB via transporter, the trace info is carried with the Context to the remote node. All nodes collect the spans locally and send to the exporters

Here is the simple generated demo where you can play with Moleculer: https://github.com/moleculerjs/project-demo

If you need any more info, just ask me.