google-cloud-dotnet: Tracing not working in GKE with Google.Cloud.Diagnostics.AspNetCore 3.0.0-beta13; works locally

Purpose Use Google Cloud Diagnostics on a .net core REST API, for Logging, Tracing and Error Reporting in two possible scenarios:

  1. Local execution on Visual Studio 2017
  2. Deployed on a Docker Container and Running on a GCP Kubernetes Engine

Description For configuring Google Cloud Diagnostics, two documentation sources were used:

  1. Google.Cloud.Diagnostics.AspNetCore
  2. Enable configuration support for Google.Cloud.Diagnostics.AspNetCore #2435

Based on the above documentation the UseGoogleDiagnostics extension method on IWebHostBuilder was used, as this configures Logging, Tracing and Error Reporting middleware.

According to the 2) link, the following table presents the information needed when using the UseGoogleDiagnostics method: Configuration Table

  1. For local execution => project_id, module_id, and version_id are needed,
  2. For GKE => module_id, and version_id

The .net core configuration files were used to provide the above information for each deployment: appsettings.json

{
  "GCP": {
    "ServiceID": "my-service",
    "VersionID": "v1"
  } 
}

appsettings.Development.json

{
  "GCP": {
    "ID": "my-id"
  } 
}

Basically, the above will render the following configuration:

  1. On Local execution
return WebHost.CreateDefaultBuilder(args)
	.UseGoogleDiagnostics("my-id", "my-service", "v1")
	.UseStartup<Startup>();
  1. On GKE
return WebHost.CreateDefaultBuilder(args)
	.UseGoogleDiagnostics(null, "my-service", "v1")
	.UseStartup<Startup>();

To guarantee i’m using the correct information, i used two places on the GCP UI to verify:

  1. On Endpoints Listing, checked the service details:
Service name: my-service Active version: v1 
  1. Checked the Endpoint logs, for a specific API POST Endpoint
{
insertId: "e6a63a28-1451-4132-ad44-a4447c33a4ac@a1"  
 
jsonPayload: {…}  
 logName: "projects/xxx%2Fendpoints_log"  
 receiveTimestamp: "2019-07-11T21:03:34.851569606Z"  
 
resource: {
  
labels: {
   location: "us-central1-a"    
   method: "v1.xxx.ApiOCRPost"    
   project_id: "my-id"    
   service: "my-service"    
   version: "v1"    
  }
  type: "api"   
 }
 severity: "INFO"  
 timestamp: "2019-07-11T21:03:27.397632588Z"  
} 

Is there a BUG? When executing the service Endpoints, for each specific deployment, Google Cloud Diagnostics behaves differently:

  1. On Local execution (VS2017) => Logging, Tracing and Error Reporting work as expected, everything showing in GCP UI
  2. On GKE Deployment => Logging, Tracing and Error Reporting DO NOT Work, nothing shows in GCP UI

I’ve tried several variations, hardcoding the values directly in the code, etc, but no matter what i do, Google Cloud Diagnostics is not working when deployed in GKE:

  1. Hardcoding the values directly
return WebHost.CreateDefaultBuilder(args)
	.UseGoogleDiagnostics(null, "my-service", "v1")
	.UseStartup<Startup>();
  1. Without the v in Version
return WebHost.CreateDefaultBuilder(args)
	.UseGoogleDiagnostics(null, "my-service", "1")
	.UseStartup<Startup>();

Am i doing anything wrong or is it a bug on Google.Cloud.Diagnostics.AspNetCore 3.0.0-beta13?

Environment details

  • .NET version: 2.2.0
  • Package name and version: Google.Cloud.Diagnostics.AspNetCore 3.0.0-beta13

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 45

Most upvoted comments

Hooray! I’m really glad we got there in the end. Thanks for all your patience 😃

I’ve finished work for the day, but I’ll look tomorrow. I’m aware of some issues with resource detection in GKE (as linked earlier) but I don’t know whether that’s what’s going wrong. Will look into it more. I’m glad it’s working when everything is specified though.

Okay - it was a long-shot, but at least it’s good to have ruled it out. Will come up with something else for you to try tomorrow 😃

Okay, investigation log…

  • Create a new solution in VS2019
  • Create a new project, selecting ASP.NET Core, then “API” template using ASP.NET Core 2.1, without configuring for HTTPS or Docker (just for simplicity).
  • Set the default log level to Information in appsettings.json so we can see a log entry for each request
  • Deploy to GKE via Cloud Build (run “dotnet publish” locally, then upload results with a Dockerfile and Cloud Build config that builds the docker image, publishes it and update the GKE deployment). I can share these files if necessary, but they shouldn’t be relevant.
  • Visit (address)/api/values - yes, it’s working
  • Visit the Log Viewer - the logs are shown under “GKE Container” -> “diagnostics-test” (the name of my GKE service) - these come from the console
  • In ValuesController.Get(int id), add code to throw an exception if the input is negative
  • Redeploy, and visit /api/values/-10
    • Error is shown in the logs, but only as info; nothing in error reporting
  • Add dependency to Google.Cloud.Diagnostics.AspNetCore 3.0.0-beta13
  • Add .UseGoogleDiagnostics(projectId, "diagnostics-test", "v1") to Program.cs
  • Redeploy, and visit /api/values
  • Visit the Log Viewer - the logs are now shown under “GKE Container” -> “aspnetcore” (the default log name)
  • Visit /api/values/-20 and check the logs: this time there’s an error log level, and the error is logged to Error Reporting

So, it looks like it’s working for me when all the values are specified, but:

  • The default log name is aspnetcore (which I agree isn’t ideal; I want to improve that)
  • I haven’t tried without specifying the various fields yet; I suspect that autodetection of the project ID may run foul of this issue

Let’s try to get your application working to the same level as my test app, and then we can make progress from there together.

Will look in detail on Monday.