google-cloud-dotnet: Tracing does not work well in Cloud Run with TimedBuffer
I have a Cloud Run service behind Cloud Load Balancer. The service is a .NET Core API, utilizing this library to log traces.
If I send 5 concurrent requests, only one gets traced. This is confirmed by checking the Trace List in the Cloud Console, and also the HTTP responses - only the first response has a X-Cloud-Trace-Context header. If I send the 5 requests, but wait for the previous one to complete before sending the next, all 5 get traced. I looked through the options and I tried specifying BufferOptions.NoBuffer() in UseGoogleDiagnostics, but that didn’t seem to make a difference. Am I missing something, or could this be a bug?
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 40 (39 by maintainers)
Commits related to this issue
- Adds documentation about running on background activities limiting environments. Closes #5275 — committed to amanda-tarafa/google-cloud-dotnet by amanda-tarafa 4 years ago
- Adds documentation about environments that limit background activities. Closes #5275 — committed to amanda-tarafa/google-cloud-dotnet by amanda-tarafa 4 years ago
- Adds documentation about environments that limit background activities. Closes #5275 — committed to amanda-tarafa/google-cloud-dotnet by amanda-tarafa 4 years ago
- Adds documentation about environments that limit background activities. Closes #5275 — committed to amanda-tarafa/google-cloud-dotnet by amanda-tarafa 4 years ago
- Release Google.Cloud.Diagnostics packages version 4.2.0-beta01 Changes in this release: - [Commit 5710321](https://github.com/googleapis/google-cloud-dotnet/commit/5710321): feat: Adds GoogleTracePro... — committed to amanda-tarafa/google-cloud-dotnet by amanda-tarafa 4 years ago
- Release Google.Cloud.Diagnostics packages version 4.2.0-beta01 Changes in this release: - [Commit 5710321](https://github.com/googleapis/google-cloud-dotnet/commit/5710321): feat: Adds GoogleTracePro... — committed to googleapis/google-cloud-dotnet by amanda-tarafa 4 years ago
There’s no way that you can manually flush the buffer right now. I could expose something via standard dependency injection that would allow you to, from a request. I don’t think it will available on SIGTERM handling though and I wouldn’t do any explicit work for that to be available on SIGTERM. I really really think you don’t need to handle SIGTERM and at that point we won’t guarantee that the Diagnostics library works, some of our dependencies or even our own objects might have been disposed off at that point, etc. All the buffers will flush on Dispose, and Dispose will happen before the application is shut down.
About your solution though, I think it won’t solve your problems, you don’t know which of the Cloud Run instances your special request will hit, there’s no guarantee on the distribution of those hits across your instances, etc. So, you will almost certainly will see delays in tracing, will run into the same problems re the buffer being very big and getting very full if for some reason one of the instances is never hit, etc.
The problem here is that you really want to do something that it’s not supported by the environment you are running in.
Just a heads up, a couple other things have come up, but this is still on my TODO for this week. I won’t be looking at it today though.
We’ll look at this on Tuesday - Monday is a public holiday in the UK.