sentry-dotnet: Automatic SQL Instrumentation Unable to Find Spans

Package

Sentry

.NET Flavor

.NET

.NET Version

6.0.0

OS

Windows

SDK Version

6.0.203

Self-Hosted Sentry Version

No response

Steps to Reproduce

I’m facing the issue originally reported in #1210 and was asked to spin up a new ticket. Upon upgrading to the latest version of Sentry, we started seeing a large number of warning log messages from sentry in the form below. Since we had jumped a few versions, I bisected until I found that these warnings started appearing in 3.22.0. I did not see these warning logs in 3.21.0. I can also confirm that disabling automatic SQL instrumentation with DisableDiagnosticSourceIntegration does remove the warning logs.

Trying to get a span of type {type} with operation id {guid}, but it was not found.

Here is an excerpt from our logs that show the sentry warnings. I’m not sure if there is some race condition, because when I set the log level to Trace (Serilog calls this Verbose), I don’t see the sentry warnings, but when the log level is set to Debug, I do.

{"Timestamp":"2022-11-18T10:09:24.3750643-05:00","Level":"Information","MessageTemplate":"Entity Framework Core {version} initialized '{contextType}' using provider '{provider}:{providerVersion}' with options: {options}","Properties":{"version":"6.0.9","contextType":"[REDACTED]","provider":"Microsoft.EntityFrameworkCore.SqlServer","providerVersion":"6.0.9","options":"None","EventId":{"Id":10403,"Name":"Microsoft.EntityFrameworkCore.Infrastructure.ContextInitialized"},"SourceContext":"Microsoft.EntityFrameworkCore.Infrastructure","ActionId":"1e01ba7b-0cc1-4926-82a9-3f9b9b60bec9","ActionName":"[REDACTED]","RequestId":"0HMM9ER7LRNIR:0000000F","RequestPath":"[REDACTED]","ConnectionId":"0HMM9ER7LRNIR","MachineName":"[REDACTED]","ClientIp":"127.0.0.1","ClientAgent":"[REDACTED]"}}
{"Timestamp":"2022-11-18T10:09:24.6612204-05:00","Level":"Warning","MessageTemplate":"Trying to get a span of type {0} with operation id {1}, but it was not found.","Properties":{"0":"Execution","1":"07546a94-0989-4d99-b5f7-c1782184eb80","SourceContext":"Sentry.ISentryClient","ActionId":"1e01ba7b-0cc1-4926-82a9-3f9b9b60bec9","ActionName":"[REDACTED]","RequestId":"0HMM9ER7LRNIR:0000000F","RequestPath":"[REDACTED]","ConnectionId":"0HMM9ER7LRNIR","MachineName":"[REDACTED]","ClientIp":"127.0.0.1","ClientAgent":"[REDACTED]"}}
{"Timestamp":"2022-11-18T10:09:24.6699988-05:00","Level":"Information","MessageTemplate":"Executed DbCommand ({elapsed}ms) [Parameters=[{parameters}], CommandType='{commandType}', CommandTimeout='{commandTimeout}']{newLine}{commandText}","Properties":{"elapsed":"56","parameters":"","commandType":"Text","commandTimeout":30,"newLine":"\r\n","commandText":"SELECT [REDACTED]\r\nFROM [REDACTED]\r\nWHERE [REDACTED]","EventId":{"Id":20101,"Name":"Microsoft.EntityFrameworkCore.Database.Command.CommandExecuted"},"SourceContext":"Microsoft.EntityFrameworkCore.Database.Command","ActionId":"1e01ba7b-0cc1-4926-82a9-3f9b9b60bec9","ActionName":"REDACTED"","RequestId":"0HMM9ER7LRNIR:0000000F","RequestPath":"[REDACTED]","ConnectionId":"0HMM9ER7LRNIR","MachineName":"[REDACTED]","ClientIp":"127.0.0.1","ClientAgent":"[REDACTED]"}}
{"Timestamp":"2022-11-18T10:09:24.7338833-05:00","Level":"Warning","MessageTemplate":"Trying to get a span of type {0} with operation id {1}, but it was not found.","Properties":{"0":"Connection","1":"bcfa705d-1d03-4c85-8ecf-16810e17ea6e","SourceContext":"Sentry.ISentryClient","ActionId":"1e01ba7b-0cc1-4926-82a9-3f9b9b60bec9","ActionName":"REDACTED"","RequestId":"0HMM9ER7LRNIR:0000000F","RequestPath":"[REDACTED]","ConnectionId":"0HMM9ER7LRNIR","MachineName":"[REDACTED]","ClientIp":"127.0.0.1","ClientAgent":"[REDACTED]"}}

Expected Result

I would not expect to see the warning log messages when using entity framework and automatic sql instrumentation enabled with Sentry.

Actual Result

An excessive number of warning log messages.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

I can confirm that with the 3.25.0 release I am no longer seeing these warning messages. Thank you for you help.

BTW - The .NET SDK guidance about TraceSampleRate needs to be updated. In general, the best results now come from setting 1.0 and relying on Sentry’s Dynamic Sampling feature.

I found the cause of the issue. In our development environments, we set the TracesSampleRate to a value less than 1. I’m guessing that you can’t find the traces because they weren’t captured. I think this probably does lend some credence to moving the messages to a Debug and not a Warning.

After much investigation, I was able to find the cause of the connection span warning. A fix is forthcoming in #2068. See there for details.

I wasn’t able to reproduce the execution span warning, but I suspect it is associated with the same problem. If it persists after this change, we can re-open the issue but I’ll need a better repro.

Thanks.