opentelemetry-dotnet: OpenTracingShim generate span with invalid spanContext for custom instrumentaion

Bug Report

List of NuGet packages and version that you are using:

  • OpenTelemetry.Shims.OpenTracing 1.0.0-rc8
  • OpenTelemetry.Extensions.Hosting 1.0.0-rc8
  • OpenTelemetry.Exporter.OpenTelemetryProtocol 1.2.0-rc1
  • OpenTracing.Contrib.NetCore 0.8.0

Runtime version: netcoreapp3.1

Symptom

We are using our own instrumentations/diagnostics to build spans for specific activities. Static TracerShim:

var openTelemetry = Sdk.CreateTracerProviderBuilder()
                .AddSource("Test.Tracing")
                .SetResourceBuilder(ResourceBuilder.CreateDefault()
                    .AddService(serviceName: "Test_ServiceName", serviceVersion: "Test_Version"))
                .AddOtlpExporter(opt =>
                {
                    opt.Endpoint = new Uri(INTERNAL_URI);
                    opt.Headers = "TOKEN=INTERNAL_TOKEN";
                })
                .Build();

            return new TracerShim(openTelemetry.GetTracer("Test.Tracing"), Propagators.DefaultTextMapPropagator);
        }

Custom diagnostics:

case "Microsoft.AspNetCore.Mvc.BeforeAction":
                    {
                        var httpContext = (HttpContext)_BeforeActionHttpContextFetcher.Fetch(untypedArg);
                        var request = httpContext.Request;
                        ISpanContext extractedSpanContext =  Tracer.Extract(BuiltinFormats.HttpHeaders,
                                new RequestHeadersExtractAdapter(request.Headers));

                        var operationName = _Options.Hosting.HttpOperationNameResolver(httpContext);
                        var actionScope = Tracer.BuildSpan(operationName)
                            .AsChildOf(extractedSpanContext)
                            .WithTag(TracingTags.HttpMethod, request.Method)
                            .WithTag(TracingTags.SpanKind, TracingTags.SpanKindServer)
                            .WithTag(TracingTags.Scheme, request.Scheme)
                            .WithTag(TracingTags.Path, request.Path.Value)
                            .WithTag(TracingTags.Host, request.Host.Value) 
                            .StartActive();

                        httpContext.Items[ActionScopeItemsKey] = actionScope;
                    }
                    break;

What is the expected behavior? The generated span will have valid spanContext in SpanContextShim. And related spans can be created successfully.

What is the actual behavior? The span contains empty fields for context which will cause Passed span's context is not valid ArgumentException when build other child spans for other activities. And no span get created successfully.

Printed span info:

"rootSpan": {
  "context": {
   "spanContext": {
    "traceId": {},
    "spanId": {},
    "traceFlags": 0,
    "isRemote": false,
    "isValid": true,
    "traceState": []
   },
   "traceId": "0828ca1eb896547a61504c4ffcfbd397",
   "spanId": "84647169dea59879"
  }
...
 }

Reproduce

Since we are leveraging same design as OpenTracing.Contrib.NetCore, we can use the project directly with opentelemetry-dotnet/examples/AspNetCore for reproducing the issue. <PackageReference Include="OpenTracing.Contrib.NetCore" Version="0.8.0" />

In Startup.cs:

services.AddOpenTelemetryTracing((builder) => builder
                    .AddSource("Test.Tracing")
                    .SetResourceBuilder(ResourceBuilder.CreateDefault()
                        .AddService(serviceName: "test-name", serviceVersion: "test-version"))
                    .AddOtlpExporter(opt =>
                    {
                        opt.Endpoint = new Uri(INTERNAL_URI);
                        opt.Headers = "TOKEN=INTERNAL_TOKEN";
                    })
            );
            services.AddSingleton<ITracer>(provider =>
            {
                var traceProvider = provider.GetRequiredService<TracerProvider>();
                var tracer = traceProvider.GetTracer("Test.Tracing");
                var tracerShim = new TracerShim(tracer, Propagators.DefaultTextMapPropagator);
                GlobalTracer.Register(tracerShim);
                return tracerShim;
            });
            services.AddOpenTracing();

Warning when running the program:

warn: OpenTracing.Contrib.NetCore.AspNetCore.AspNetCoreDiagnostics[0]
      Event-Exception: Microsoft.AspNetCore.Mvc.BeforeActionResult
System.ArgumentException: Passed span's context is not valid (Parameter 'Context')
   at OpenTelemetry.Shims.OpenTracing.SpanShim..ctor(TelemetrySpan span)
   at OpenTelemetry.Shims.OpenTracing.SpanBuilderShim.Start()
   at OpenTelemetry.Shims.OpenTracing.SpanBuilderShim.StartActive(Boolean finishSpanOnDispose)
   at OpenTelemetry.Shims.OpenTracing.SpanBuilderShim.StartActive()
   at OpenTracing.Contrib.NetCore.AspNetCore.AspNetCoreDiagnostics.HandleEvent(String eventName, Object untypedArg)
   at OpenTracing.Contrib.NetCore.Internal.DiagnosticEventObserver.System.IObserver<System.Collections.Generic.KeyValuePair<System.String,System.Object>>.OnNext(KeyValuePair`2 value)

Additional Context

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 4
  • Comments: 17 (7 by maintainers)

Most upvoted comments

@sasha-khadasevich, fix from #4668 should be included in next 1.6.0-something release.

Other parts related to

are not fixed yet there is no ETA.

Hi @yehaotian, I was on a short vacation. I will debug this issue this week or the next.

I still get error when DO NOT have .AddAspNetCoreInstrumentation(), what could be the root cause?

The problem is that OpenTracing shim is broken if there is no parent activity. It is related to other problem in the sense that they hit the same exception, but, likely need two different fixes: one for “no listeners” and another to correctly handle root span.

I will take a look at the code to see if I can quickly fix both issues.

Also does “OpenTracing” have to be the source name?

No, you should be able to customize it.

Thanks for the response, will try find some time to update the environment and do the test