google-cloud-dotnet: Tracing not working in GKE with Google.Cloud.Diagnostics.AspNetCore 3.0.0-beta13; works locally
Purpose Use Google Cloud Diagnostics on a .net core REST API, for Logging, Tracing and Error Reporting in two possible scenarios:
- Local execution on Visual Studio 2017
- Deployed on a Docker Container and Running on a GCP Kubernetes Engine
Description For configuring Google Cloud Diagnostics, two documentation sources were used:
- Google.Cloud.Diagnostics.AspNetCore
- Enable configuration support for Google.Cloud.Diagnostics.AspNetCore #2435
Based on the above documentation the UseGoogleDiagnostics extension method on IWebHostBuilder was used, as this configures Logging, Tracing and Error Reporting middleware.
According to the 2) link, the following table presents the information needed when using the UseGoogleDiagnostics method:
- For local execution => project_id, module_id, and version_id are needed,
- For GKE => module_id, and version_id
The .net core configuration files were used to provide the above information for each deployment: appsettings.json
{
"GCP": {
"ServiceID": "my-service",
"VersionID": "v1"
}
}
appsettings.Development.json
{
"GCP": {
"ID": "my-id"
}
}
Basically, the above will render the following configuration:
- On Local execution
return WebHost.CreateDefaultBuilder(args)
.UseGoogleDiagnostics("my-id", "my-service", "v1")
.UseStartup<Startup>();
- On GKE
return WebHost.CreateDefaultBuilder(args)
.UseGoogleDiagnostics(null, "my-service", "v1")
.UseStartup<Startup>();
To guarantee i’m using the correct information, i used two places on the GCP UI to verify:
- On Endpoints Listing, checked the service details:
Service name: my-service Active version: v1
- Checked the Endpoint logs, for a specific API POST Endpoint
{
insertId: "e6a63a28-1451-4132-ad44-a4447c33a4ac@a1"
jsonPayload: {…}
logName: "projects/xxx%2Fendpoints_log"
receiveTimestamp: "2019-07-11T21:03:34.851569606Z"
resource: {
labels: {
location: "us-central1-a"
method: "v1.xxx.ApiOCRPost"
project_id: "my-id"
service: "my-service"
version: "v1"
}
type: "api"
}
severity: "INFO"
timestamp: "2019-07-11T21:03:27.397632588Z"
}
Is there a BUG? When executing the service Endpoints, for each specific deployment, Google Cloud Diagnostics behaves differently:
- On Local execution (VS2017) => Logging, Tracing and Error Reporting work as expected, everything showing in GCP UI
- On GKE Deployment => Logging, Tracing and Error Reporting DO NOT Work, nothing shows in GCP UI
I’ve tried several variations, hardcoding the values directly in the code, etc, but no matter what i do, Google Cloud Diagnostics is not working when deployed in GKE:
- Hardcoding the values directly
return WebHost.CreateDefaultBuilder(args)
.UseGoogleDiagnostics(null, "my-service", "v1")
.UseStartup<Startup>();
- Without the v in Version
return WebHost.CreateDefaultBuilder(args)
.UseGoogleDiagnostics(null, "my-service", "1")
.UseStartup<Startup>();
Am i doing anything wrong or is it a bug on Google.Cloud.Diagnostics.AspNetCore 3.0.0-beta13?
Environment details
- .NET version: 2.2.0
- Package name and version: Google.Cloud.Diagnostics.AspNetCore 3.0.0-beta13
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 45
Hooray! I’m really glad we got there in the end. Thanks for all your patience 😃
I’ve finished work for the day, but I’ll look tomorrow. I’m aware of some issues with resource detection in GKE (as linked earlier) but I don’t know whether that’s what’s going wrong. Will look into it more. I’m glad it’s working when everything is specified though.
Okay - it was a long-shot, but at least it’s good to have ruled it out. Will come up with something else for you to try tomorrow 😃
Okay, investigation log…
ValuesController.Get(int id)
, add code to throw an exception if the input is negative.UseGoogleDiagnostics(projectId, "diagnostics-test", "v1")
to Program.csSo, it looks like it’s working for me when all the values are specified, but:
Let’s try to get your application working to the same level as my test app, and then we can make progress from there together.
Will look in detail on Monday.