sentry-dotnet: Automatic SQL Instrumentation Unable to Find Spans
Package
Sentry
.NET Flavor
.NET
.NET Version
6.0.0
OS
Windows
SDK Version
6.0.203
Self-Hosted Sentry Version
No response
Steps to Reproduce
I’m facing the issue originally reported in #1210 and was asked to spin up a new ticket. Upon upgrading to the latest version of Sentry, we started seeing a large number of warning log messages from sentry in the form below. Since we had jumped a few versions, I bisected until I found that these warnings started appearing in 3.22.0. I did not see these warning logs in 3.21.0. I can also confirm that disabling automatic SQL instrumentation with DisableDiagnosticSourceIntegration does remove the warning logs.
Trying to get a span of type {type} with operation id {guid}, but it was not found.
Here is an excerpt from our logs that show the sentry warnings. I’m not sure if there is some race condition, because when I set the log level to Trace (Serilog calls this Verbose), I don’t see the sentry warnings, but when the log level is set to Debug, I do.
{"Timestamp":"2022-11-18T10:09:24.3750643-05:00","Level":"Information","MessageTemplate":"Entity Framework Core {version} initialized '{contextType}' using provider '{provider}:{providerVersion}' with options: {options}","Properties":{"version":"6.0.9","contextType":"[REDACTED]","provider":"Microsoft.EntityFrameworkCore.SqlServer","providerVersion":"6.0.9","options":"None","EventId":{"Id":10403,"Name":"Microsoft.EntityFrameworkCore.Infrastructure.ContextInitialized"},"SourceContext":"Microsoft.EntityFrameworkCore.Infrastructure","ActionId":"1e01ba7b-0cc1-4926-82a9-3f9b9b60bec9","ActionName":"[REDACTED]","RequestId":"0HMM9ER7LRNIR:0000000F","RequestPath":"[REDACTED]","ConnectionId":"0HMM9ER7LRNIR","MachineName":"[REDACTED]","ClientIp":"127.0.0.1","ClientAgent":"[REDACTED]"}}
{"Timestamp":"2022-11-18T10:09:24.6612204-05:00","Level":"Warning","MessageTemplate":"Trying to get a span of type {0} with operation id {1}, but it was not found.","Properties":{"0":"Execution","1":"07546a94-0989-4d99-b5f7-c1782184eb80","SourceContext":"Sentry.ISentryClient","ActionId":"1e01ba7b-0cc1-4926-82a9-3f9b9b60bec9","ActionName":"[REDACTED]","RequestId":"0HMM9ER7LRNIR:0000000F","RequestPath":"[REDACTED]","ConnectionId":"0HMM9ER7LRNIR","MachineName":"[REDACTED]","ClientIp":"127.0.0.1","ClientAgent":"[REDACTED]"}}
{"Timestamp":"2022-11-18T10:09:24.6699988-05:00","Level":"Information","MessageTemplate":"Executed DbCommand ({elapsed}ms) [Parameters=[{parameters}], CommandType='{commandType}', CommandTimeout='{commandTimeout}']{newLine}{commandText}","Properties":{"elapsed":"56","parameters":"","commandType":"Text","commandTimeout":30,"newLine":"\r\n","commandText":"SELECT [REDACTED]\r\nFROM [REDACTED]\r\nWHERE [REDACTED]","EventId":{"Id":20101,"Name":"Microsoft.EntityFrameworkCore.Database.Command.CommandExecuted"},"SourceContext":"Microsoft.EntityFrameworkCore.Database.Command","ActionId":"1e01ba7b-0cc1-4926-82a9-3f9b9b60bec9","ActionName":"REDACTED"","RequestId":"0HMM9ER7LRNIR:0000000F","RequestPath":"[REDACTED]","ConnectionId":"0HMM9ER7LRNIR","MachineName":"[REDACTED]","ClientIp":"127.0.0.1","ClientAgent":"[REDACTED]"}}
{"Timestamp":"2022-11-18T10:09:24.7338833-05:00","Level":"Warning","MessageTemplate":"Trying to get a span of type {0} with operation id {1}, but it was not found.","Properties":{"0":"Connection","1":"bcfa705d-1d03-4c85-8ecf-16810e17ea6e","SourceContext":"Sentry.ISentryClient","ActionId":"1e01ba7b-0cc1-4926-82a9-3f9b9b60bec9","ActionName":"REDACTED"","RequestId":"0HMM9ER7LRNIR:0000000F","RequestPath":"[REDACTED]","ConnectionId":"0HMM9ER7LRNIR","MachineName":"[REDACTED]","ClientIp":"127.0.0.1","ClientAgent":"[REDACTED]"}}
Expected Result
I would not expect to see the warning log messages when using entity framework and automatic sql instrumentation enabled with Sentry.
Actual Result
An excessive number of warning log messages.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (9 by maintainers)
I can confirm that with the 3.25.0 release I am no longer seeing these warning messages. Thank you for you help.
BTW - The .NET SDK guidance about
TraceSampleRateneeds to be updated. In general, the best results now come from setting1.0and relying on Sentry’s Dynamic Sampling feature.I found the cause of the issue. In our development environments, we set the TracesSampleRate to a value less than 1. I’m guessing that you can’t find the traces because they weren’t captured. I think this probably does lend some credence to moving the messages to a Debug and not a Warning.
After much investigation, I was able to find the cause of the connection span warning. A fix is forthcoming in #2068. See there for details.
I wasn’t able to reproduce the execution span warning, but I suspect it is associated with the same problem. If it persists after this change, we can re-open the issue but I’ll need a better repro.
Thanks.