aspire: Container telemetry not shown in AppHost dashboard

In preview 5, adding .WithOtlpExporter() to .AddContainer(...) would show OpenTelemetry logs, traces and metrics in the dashboard.

Since preview 6 this doesn’t happen.

But if I had a dashboard container (mcr.microsoft.com/dotnet/aspire-dashboard or mcr.microsoft.com/dotnet/nightly/aspire-dashboard:8.0-preview):

var aspireDashboard = builder.AddContainer(
    "aspire-dashboard",
    "mcr.microsoft.com/dotnet/aspire-dashboard",
    "8.0-preview")
    .WithEndpoint(targetPort: 18888, port: 5018, name: "dash", scheme: "http", isProxied: true)
    .WithEndpoint(targetPort: 18889, port: 5019, name: "otel", scheme: "http", isProxied: true)
    .WithEnvironment("DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS", "true")
    .WithEnvironment("ASPIRE_ALLOW_UNSECURED_TRANSPORT", "true")
    .WithEnvironment("DASHBOARD__OTLP__AUTHMODE", "ApiKey")
    .WithEnvironment("DASHBOARD__OTLP__PRIMARYAPIKEY", builder.Configuration["AppHost:OtlpApiKey"])
    ;
var service = builder.AddContainer(...)
    .WithEndpoint(targetPort: 8080, port: 5012, name: "http", scheme: "http")
    .WithEnvironment("OTEL_BLRP_SCHEDULE_DELAY", "1000")
    .WithEnvironment("OTEL_BSP_SCHEDULE_DELAY", "1000")
    .WithEnvironment("OTEL_METRIC_EXPORT_INTERVAL", "1000")
    .WithEnvironment("OTEL_TRACES_SAMPLER", "always_on")
    .WithEnvironment("OTEL_EXPORTER_OTLP_PROTOCOL", "grpc")
    .WithEnvironment("OTEL_SERVICE_NAME", "hub")
    .WithEnvironment("OTEL_RESOURCE_ATTRIBUTES", "service.instance.id=hub")
    .WithEnvironment("OTEL_EXPORTER_OTLP_HEADERS", $"x-otlp-api-key={builder.Configuration["AppHost:OtlpApiKey"]}")
    .WithEnvironment(ctx => ctx.EnvironmentVariables["OTEL_EXPORTER_OTLP_ENDPOINT"] = new HostUrl(hubAspireDashboard.GetEndpoint("otel").Url))
...

the dashboard from the aspireDashboard container show telemetry from the service container.

service container is referenced from other resources and references other resources without any issues.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 48 (39 by maintainers)

Most upvoted comments

OK so the issue here is that when containers want to talk to services running on the host machine (rather than other containers) they do so via a special host name, host.docker.internal or host.containers.internal (depending on whether you’re using Docker or Podman). The Aspire Dashboard is running on the host machine and is hosting the OTLP endpoint via HTTPS. The cerficiate used for the HTTPS endpoint is the ASP.NET Core HTTPS dev cert, the same one used by all ASP.NET Core apps by default during development. This self-signed certificate is generally trusted on the host machine as part of the .NET inner-loop (either by VS/C#DK prompting you to trust it, or by running dotnet dev-certs https --trust manually), but it won’t be trusted by processes running in containers, or executables that don’t use the hosting machine’s certificate store/infra for cert operations (e.g. Node.js). Additionally, the certificate is only valid for the localhost host name (i.e. that’s the certificate subject).

So to make this work we need two things:

  1. The dev cert needs to have additional SANs (subject alternative names) for the special container network host names (host.docker.internal and host.containers.internal). In the future it’s possible we’ll expand this to other special reserved local-only host names like *.localhost to enable scenarios like #5508
  2. The dev cert needs to be trusted by any clients trying to access a service via an HTTPS endpoint using that cert

For 1, we’re planning on updating dotnet dev-certs in a .NET SDK update so that the dev cert includes the SANs described. We’d likely do that in a servicing release of the 8.0.xxx and 9.0.xxx SDKs. The dev cert infrastructure includes a mechanism to version the cert so that we can make changes like this and the new cert will be laid down for you when you update your SDK. You’ll need to re-trust it of course but if you’re using VS or VS Code with C# Dev Kit you’ll be prompted to do so the first time you launch after updating your SDK, otherwise you’ll need to run dotnet dev-certs https --trust yourself (like usual).

For 2, we’ll productize the capability from our samples repo that facilitates injecting the dev cert into Aspire resources (containers or executables) by exporting it from the host (using dotnet dev-certs https --export-path), setting an environment variable on the resource pointing to the cert file(s), and, in the case of containers, bind-mounting it into the container. Then each hosting integration can be updated to use this capability and do whatever other configuration is required to configure the specific container/executable to use and/or trust the dev cert (e.g. Node.js requires setting the NODE_EXTRA_CA_CERTS environment variable to point to the paths of certificate files that you want to be trusted beyond the usual public root CAs). (Related dotnet/aspnetcore#60369)

We’d like to be able to get all this done in an Aspire 9.x update. Soonest would be 9.2 but that’s optimistic.

RE how flexible the cert injection will be, you’ll be able to customize the environment variables and bind mount path. If you want to take over completely, you can simply call an API to request the dev cert file path (Pem or Pfx format) and then take full control of how you want to inject it into the container.

RE custom certificates, seems like a nice complimentary feature. For ASP.NET Core apps you can already specify the path to a cert on disk to use (via Kestrel configuration) so that shouldn’t be a problem. Making that first class is something I could see us doing.

We know and are working on it. It’s not simple and requires changes to various parts of the stack. This is part of a bigger “make https work for containers”

Using an unencrypted transport for OTEL means processes running as other accounts on the same machine during local development can potentially observe sensitive information being passed from apps to the dashboard’s OTLP endpoint so we decided that behavior must be opt-in.

For the standalone dashboard though it is unsecured and there’s a warning in the dashboard for it.

It’s likely HTTPS as the HTTPS certificate being used to secure the dashboard endpoints when it’s run from the host is the ASP.NET Core Development Certificate which will not be trusted by default from inside a container. This will cause the OTLP exporter to fail. This change was introduced in preview 6 as part of security work. In the example you provided of running the dashboard itself in a container, you’re explicitly disabling the HTTPS transport via the ASPIRE_ALLOW_UNSECURED_TRANSPORT environment variable and creating only HTTP endpoints using WithEndpoint.

Having composed containers be automatically configured to trust the ASP.NET Core Development Certificate is something we want to investigate enabling in the future.

We’re working on a broader solution for this @DamianEdwards can add color.

@Depechie Try upgrading to 8.2.1 - sounds like you’re hitting #5532 .

@JamesNK should we consider adding some new docs about this scenario?

For images that are being pulled, wouldn’t forcing the container to trust a certificate that shouldn’t be trusted defeat the whole purpose of the security added to Aspire?

I don’t think so. Configuring a container to trust a self-signed certificate that is already trusted by the host for development purposes (and only at development time) seems no different to what we do today WRT to trusting the ASP.NET Core Development Certificate for the purposes of localhost HTTPS testing. The trust would only happen when running during local development as the certificate is only for localhost anyway.

Using an unencrypted transport for OTEL means processes running as other accounts on the same machine during local development can potentially observe sensitive information being passed from apps to the dashboard’s OTLP endpoint so we decided that behavior must be opt-in.

OTEL logs don’t show up in normal logs - you have to do a bit of extra magic to get them by creating an OTEL_DIAGNOSTICS.json file - https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry/README.md#self-diagnostics

I was able to reproduce with the Aspire starter demo on Rancher Desktop.

  1. Create a new starter solution: dotnet new aspire-starter.
  2. Change the ApiService to produce a container.
  3. Build and publish the container.
  4. Change the Aspire host:
var builder = DistributedApplication.CreateBuilder(args);

//var apiService = builder.AddProject<Projects.ApiService>("apiservice")
var apiService = builder.AddContainer("apiservice", "apiservice")
    .WithHttpEndpoint(targetPort: 8080, name: "apiservice")
    .WithOtlpExporter()
    ;

builder.AddProject<Projects.Web>("webfrontend")
    .WithExternalHttpEndpoints()
    .WithReference(apiService.GetEndpoint("apiservice"))
    ;

builder.Build().Run();
  1. Run the Aspire host.

You can check that all OTEL environment variables are set for the container, but no telemetry is captured by the Aspire host.