opentelemetry-dotnet-instrumentation: Instrumentation does not work with dotnet CLI

[EDIT @pjanotti: the issue is fixed for versions after [v0.6.0-beta.1(]https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/tag/v0.6.0-beta.1)]

Exception:

PS C:\Users\Administrator\Desktop\OTelConsole-NET6.0> dotnet run My.Simple.Console
Unhandled exception. System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.TypeInitializationException: The type initializer for 'OpenTelemetry.AutoInstrumentation.Loader.Startup' threw an exception.
 ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Extensions.Configuration.Abstractions, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
File name: 'Microsoft.Extensions.Configuration.Abstractions, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
   at Microsoft.Extensions.DependencyInjection.ProviderBuilderServiceCollectionExtensions.<>c.<AddOpenTelemetryProviderBuilderServices>b__2_0(IServiceProvider sp)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at OpenTelemetry.Internal.ConfigurationExtensions.<>c__DisplayClass5_0`1.<RegisterOptionsFactory>b__0(IServiceProvider sp)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at OpenTelemetry.Trace.OtlpTraceExporterHelperExtensions.<>c__DisplayClass2_0.<AddOtlpExporter>b__1(IServiceProvider sp, TracerProviderBuilder builder)
   at OpenTelemetry.ProviderBuilderServiceCollectionCallbackHelper`3.<>c__DisplayClass0_0.<RegisterConfigureBuilderCallback>b__0(IServiceProvider sp, TState state)
   at OpenTelemetry.ProviderBuilderServiceCollectionCallbackHelper`3.ConfigureProviderBuilderStateCallbackRegistration.Configure(IServiceProvider serviceProvider, TState state)
   at OpenTelemetry.ProviderBuilderServiceCollectionCallbackHelper`3.InvokeRegisteredConfigureStateCallbacks(IServiceProvider serviceProvider, TState state)
   at OpenTelemetry.Trace.TracerProviderSdk..ctor(IServiceProvider serviceProvider, Boolean ownsServiceProvider)
   at OpenTelemetry.Trace.TracerProviderBuilderBase.Build()
   at OpenTelemetry.Trace.TracerProviderBuilderBase.InvokeBuild()
   at OpenTelemetry.Trace.TracerProviderBuilderExtensions.Build(TracerProviderBuilder tracerProviderBuilder)
   at OpenTelemetry.AutoInstrumentation.Instrumentation.Initialize() in F:\Dev\forks\rassk\opentelemetry-dotnet-instrumentation\src\OpenTelemetry.AutoInstrumentation\Instrumentation.cs:line 121
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at OpenTelemetry.AutoInstrumentation.Loader.Startup.TryLoadManagedAssembly() in F:\Dev\forks\rassk\opentelemetry-dotnet-instrumentation\src\OpenTelemetry.AutoInstrumentation.Loader\Startup.cs:line 71
   at OpenTelemetry.AutoInstrumentation.Loader.Startup..cctor() in F:\Dev\forks\rassk\opentelemetry-dotnet-instrumentation\src\OpenTelemetry.AutoInstrumentation.Loader\Startup.cs:line 47
   --- End of inner exception stack trace ---
   at OpenTelemetry.AutoInstrumentation.Loader.Startup..ctor()
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions)
   --- End of inner exception stack trace ---
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic, Boolean wrapExceptions)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Reflection.Assembly.CreateInstance(String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Reflection.Assembly.CreateInstance(String typeName)
   at StartupHook.Initialize() in F:\Dev\forks\rassk\opentelemetry-dotnet-instrumentation\src\OpenTelemetry.AutoInstrumentation.StartupHook\StartupHook.cs:line 75
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at System.StartupHookProvider.CallStartupHook(StartupHookNameOrPath startupHook)
   at System.StartupHookProvider.ProcessStartupHooks()

Test App:

using System.Diagnostics;

namespace My.Simple.ConsoleApp;

internal class Program
{
    private static ActivitySource MySource = new ActivitySource("My.Simple.Console");

    public static void Main(string[] args)
    {
        Console.WriteLine("Hello My.Simple.ConsoleApp");

        using (var activity = MySource.StartActivity("Main"))
        {
            Console.WriteLine("Inside activity create");

            activity?.AddTag("my.custom.tag", "my.simple.consoleapp");
        }

        Console.WriteLine("All done");
    }
}
<Project Sdk="Microsoft.NET.Sdk">

	<PropertyGroup>
		<OutputType>exe</OutputType>
		<TargetFramework>net6.0</TargetFramework>
		<ImplicitUsings>enable</ImplicitUsings>
		<Nullable>enable</Nullable>
	</PropertyGroup>

	<ItemGroup>
		<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="7.0.0" />
		<PackageReference Include="System.Diagnostics.DiagnosticSource" Version="7.0.0" />
	</ItemGroup>

</Project>

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 19 (19 by maintainers)

Most upvoted comments

I could recreate the issue on dotnet test.

Error:
  An assembly specified in the application dependencies manifest (OpenTelemetry.AutoInstrumentation.AdditionalDeps.deps.json) was not found:
    package: 'DnsClient', version: '1.4.0'
    path: 'lib/netstandard2.1/DnsClient.dll'

Issue gets resolved, if I manually reset the environment variables to below values.

export DOTNET_STARTUP_HOOKS=/d/opentelemetry-dotnet-instrumentation/bin/tracer-home/net/OpenTelemetry.AutoInstrumentation.StartupHook.dll
export DOTNET_SHARED_STORE=/d/opentelemetry-dotnet-instrumentation/bin/tracer-home/store
export DOTNET_ADDITIONAL_DEPS=/d/opentelemetry-dotnet-instrumentation/bin/tracer-home/AdditionalDeps
export ASPNETCORE_HOSTINGSTARTUPASSEMBLIES=OpenTelemetry.AutoInstrumentation.AspNetCoreBootstrapper
export OTEL_DOTNET_AUTO_HOME=/d/opentelemetry-dotnet-instrumentation/bin/tracer-home
export OTEL_DOTNET_AUTO_TRACES_CONSOLE_EXPORTER_ENABLED=true
export OTEL_DOTNET_AUTO_METRICS_CONSOLE_EXPORTER_ENABLED=true
export OTEL_DOTNET_AUTO_LOGS_CONSOLE_EXPORTER_ENABLED=true
export OTEL_SERVICE_NAME=DemoService
export OTEL_DOTNET_AUTO_DEBUG=true
export OTEL_DOTNET_AUTO_LOG_DIRECTORY=/C/temp/

Collected dotnet trace to check why we ran into an issue. In the non-working case shared store environment variable was not honored. I don’t know the reason for it. Resetting only shared store environment variable did not solve an issue. I had to reset all the environment variables like above to get it working.

Non-Working

Found registered install location 'C:\Program Files\dotnet\'.
-- arguments_t: host_path='C:\Program Files\dotnet\dotnet.exe' app_root='C:\Program Files\dotnet\sdk\7.0.102\' deps='C:\Program Files\dotnet\sdk\7.0.102\dotnet.deps.json' core_svc='C:\Program Files (x86)\coreservicing' mgd_app='C:\Program Files\dotnet\sdk\7.0.102\dotnet.dll'
-- arguments_t: dotnet shared store: 'C:\Program Files\dotnet\store\x64\net7.0'
-- arguments_t: global shared store: 'C:\Program Files\dotnet\store\x64\net7.0'

Working

Found registered install location 'C:\Program Files\dotnet\'.
-- arguments_t: host_path='C:\Program Files\dotnet\dotnet.exe' app_root='C:\Program Files\dotnet\sdk\7.0.102\' deps='C:\Program Files\dotnet\sdk\7.0.102\dotnet.deps.json' core_svc='C:\Program Files (x86)\coreservicing' mgd_app='C:\Program Files\dotnet\sdk\7.0.102\dotnet.dll'
-- arguments_t: env shared store: 'D:\temp\Azure.OpenTelemetry\store\x64\net7.0'
-- arguments_t: dotnet shared store: 'C:\Program Files\dotnet\store\x64\net7.0'
-- arguments_t: global shared store: 'C:\Program Files\dotnet\store\x64\net7.0'

Below were the environmental variable set, just by running instrument.sh.

COR_PROFILER={918728DD-259F-4A6A-AC2B-B85E1B658318}
COR_PROFILER_PATH_64=D:\opentelemetry-dotnet-instrumentation\bin\tracer-home/win-x64/OpenTelemetry.AutoInstrumentation.Native.dll
DOTNET_SHARED_STORE=D:\opentelemetry-dotnet-instrumentation\bin\tracer-home\store
CORECLR_PROFILER={918728DD-259F-4A6A-AC2B-B85E1B658318}
COR_ENABLE_PROFILING=1
COR_PROFILER_PATH_32=D:\opentelemetry-dotnet-instrumentation\bin\tracer-home/win-x86/OpenTelemetry.AutoInstrumentation.Native.dll
OTEL_DOTNET_AUTO_INTEGRATIONS_FILE=D:\opentelemetry-dotnet-instrumentation\bin\tracer-home/integrations.json
CORECLR_PROFILER_PATH_64=D:\opentelemetry-dotnet-instrumentation\bin\tracer-home/win-x64/OpenTelemetry.AutoInstrumentation.Native.dll
CORECLR_ENABLE_PROFILING=1
DOTNET_ADDITIONAL_DEPS=D:\opentelemetry-dotnet-instrumentation\bin\tracer-home/AdditionalDeps
CORECLR_PROFILER_PATH_32=D:\opentelemetry-dotnet-instrumentation\bin\tracer-home/win-x86/OpenTelemetry.AutoInstrumentation.Native.dll
DOTNET_STARTUP_HOOKS=D:\opentelemetry-dotnet-instrumentation\bin\tracer-home/net/OpenTelemetry.AutoInstrumentation.StartupHook.dll
OTEL_DOTNET_AUTO_HOME=D:\opentelemetry-dotnet-instrumentation\bin\tracer-home
DOTNET_HOST_PATH=C:\Program Files\dotnet\dotnet.exe

Further analysis is needed to understand what is an issue with instrument.sh set environment variables.

Skip from StartupHook is working correctly regarding the excluded processes. However, it seems that we should let the StartupHook deal with DOTNET_ADDITIONAL_DEPS and DOTNET_SHARED_STORE to avoid the error above. Otherwise, it becomes impossible to use dotnet CLI. That said per my experiments we are not out of the woods even regarding dotnet run because the application itself is still crashing due to missing dependencies.

Setting this helped. $env:OTEL_DOTNET_AUTO_EXCLUDE_PROCESSES = "dotnet,dotnet.exe"

I think we should consider baking some well known exceptions into our solution.