aspire: Container telemetry not shown in AppHost dashboard
In preview 5, adding .WithOtlpExporter() to .AddContainer(...) would show OpenTelemetry logs, traces and metrics in the dashboard.
Since preview 6 this doesn’t happen.
But if I had a dashboard container (mcr.microsoft.com/dotnet/aspire-dashboard or mcr.microsoft.com/dotnet/nightly/aspire-dashboard:8.0-preview):
var aspireDashboard = builder.AddContainer(
"aspire-dashboard",
"mcr.microsoft.com/dotnet/aspire-dashboard",
"8.0-preview")
.WithEndpoint(targetPort: 18888, port: 5018, name: "dash", scheme: "http", isProxied: true)
.WithEndpoint(targetPort: 18889, port: 5019, name: "otel", scheme: "http", isProxied: true)
.WithEnvironment("DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS", "true")
.WithEnvironment("ASPIRE_ALLOW_UNSECURED_TRANSPORT", "true")
.WithEnvironment("DASHBOARD__OTLP__AUTHMODE", "ApiKey")
.WithEnvironment("DASHBOARD__OTLP__PRIMARYAPIKEY", builder.Configuration["AppHost:OtlpApiKey"])
;
var service = builder.AddContainer(...)
.WithEndpoint(targetPort: 8080, port: 5012, name: "http", scheme: "http")
.WithEnvironment("OTEL_BLRP_SCHEDULE_DELAY", "1000")
.WithEnvironment("OTEL_BSP_SCHEDULE_DELAY", "1000")
.WithEnvironment("OTEL_METRIC_EXPORT_INTERVAL", "1000")
.WithEnvironment("OTEL_TRACES_SAMPLER", "always_on")
.WithEnvironment("OTEL_EXPORTER_OTLP_PROTOCOL", "grpc")
.WithEnvironment("OTEL_SERVICE_NAME", "hub")
.WithEnvironment("OTEL_RESOURCE_ATTRIBUTES", "service.instance.id=hub")
.WithEnvironment("OTEL_EXPORTER_OTLP_HEADERS", $"x-otlp-api-key={builder.Configuration["AppHost:OtlpApiKey"]}")
.WithEnvironment(ctx => ctx.EnvironmentVariables["OTEL_EXPORTER_OTLP_ENDPOINT"] = new HostUrl(hubAspireDashboard.GetEndpoint("otel").Url))
...
the dashboard from the aspireDashboard container show telemetry from the service container.
service container is referenced from other resources and references other resources without any issues.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 48 (39 by maintainers)
OK so the issue here is that when containers want to talk to services running on the host machine (rather than other containers) they do so via a special host name,
host.docker.internalorhost.containers.internal(depending on whether you’re using Docker or Podman). The Aspire Dashboard is running on the host machine and is hosting the OTLP endpoint via HTTPS. The cerficiate used for the HTTPS endpoint is the ASP.NET Core HTTPS dev cert, the same one used by all ASP.NET Core apps by default during development. This self-signed certificate is generally trusted on the host machine as part of the .NET inner-loop (either by VS/C#DK prompting you to trust it, or by runningdotnet dev-certs https --trustmanually), but it won’t be trusted by processes running in containers, or executables that don’t use the hosting machine’s certificate store/infra for cert operations (e.g. Node.js). Additionally, the certificate is only valid for thelocalhosthost name (i.e. that’s the certificate subject).So to make this work we need two things:
host.docker.internalandhost.containers.internal). In the future it’s possible we’ll expand this to other special reserved local-only host names like*.localhostto enable scenarios like #5508For 1, we’re planning on updating
dotnet dev-certsin a .NET SDK update so that the dev cert includes the SANs described. We’d likely do that in a servicing release of the 8.0.xxx and 9.0.xxx SDKs. The dev cert infrastructure includes a mechanism to version the cert so that we can make changes like this and the new cert will be laid down for you when you update your SDK. You’ll need to re-trust it of course but if you’re using VS or VS Code with C# Dev Kit you’ll be prompted to do so the first time you launch after updating your SDK, otherwise you’ll need to rundotnet dev-certs https --trustyourself (like usual).For 2, we’ll productize the capability from our samples repo that facilitates injecting the dev cert into Aspire resources (containers or executables) by exporting it from the host (using
dotnet dev-certs https --export-path), setting an environment variable on the resource pointing to the cert file(s), and, in the case of containers, bind-mounting it into the container. Then each hosting integration can be updated to use this capability and do whatever other configuration is required to configure the specific container/executable to use and/or trust the dev cert (e.g. Node.js requires setting theNODE_EXTRA_CA_CERTSenvironment variable to point to the paths of certificate files that you want to be trusted beyond the usual public root CAs). (Related dotnet/aspnetcore#60369)We’d like to be able to get all this done in an Aspire 9.x update. Soonest would be 9.2 but that’s optimistic.
RE how flexible the cert injection will be, you’ll be able to customize the environment variables and bind mount path. If you want to take over completely, you can simply call an API to request the dev cert file path (Pem or Pfx format) and then take full control of how you want to inject it into the container.
RE custom certificates, seems like a nice complimentary feature. For ASP.NET Core apps you can already specify the path to a cert on disk to use (via Kestrel configuration) so that shouldn’t be a problem. Making that first class is something I could see us doing.
We know and are working on it. It’s not simple and requires changes to various parts of the stack. This is part of a bigger “make https work for containers”
For the standalone dashboard though it is unsecured and there’s a warning in the dashboard for it.
It’s likely HTTPS as the HTTPS certificate being used to secure the dashboard endpoints when it’s run from the host is the ASP.NET Core Development Certificate which will not be trusted by default from inside a container. This will cause the OTLP exporter to fail. This change was introduced in preview 6 as part of security work. In the example you provided of running the dashboard itself in a container, you’re explicitly disabling the HTTPS transport via the
ASPIRE_ALLOW_UNSECURED_TRANSPORTenvironment variable and creating only HTTP endpoints usingWithEndpoint.Having composed containers be automatically configured to trust the ASP.NET Core Development Certificate is something we want to investigate enabling in the future.
We’re working on a broader solution for this @DamianEdwards can add color.
@Depechie Try upgrading to 8.2.1 - sounds like you’re hitting #5532 .
@JamesNK should we consider adding some new docs about this scenario?
I don’t think so. Configuring a container to trust a self-signed certificate that is already trusted by the host for development purposes (and only at development time) seems no different to what we do today WRT to trusting the ASP.NET Core Development Certificate for the purposes of localhost HTTPS testing. The trust would only happen when running during local development as the certificate is only for localhost anyway.
Using an unencrypted transport for OTEL means processes running as other accounts on the same machine during local development can potentially observe sensitive information being passed from apps to the dashboard’s OTLP endpoint so we decided that behavior must be opt-in.
OTEL logs don’t show up in normal logs - you have to do a bit of extra magic to get them by creating an
OTEL_DIAGNOSTICS.jsonfile - https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry/README.md#self-diagnosticsI was able to reproduce with the Aspire starter demo on Rancher Desktop.
dotnet new aspire-starter.ApiServiceto produce a container.You can check that all OTEL environment variables are set for the container, but no telemetry is captured by the Aspire host.