sentry-python: Profiles will not work with a `traces_sampler` instead of `traces_sample_rate`

How do you use Sentry?

Sentry Saas (sentry.io)

Version

1.12.1

Steps to Reproduce

Pass a TracesSampler instead of setting traces_sample_rate
Set _experiments['profiles_sample_rate'] to 1.0

Expected Result

Every request with a trace also has a corresponding profile

Actual Result

No profiles are being recorded

P.S. From glancing at the code it doesn’t look like there is a hard dependency on the more “dumb” traces_sample_rate over a TracesSampler. It’s just that init checks for traces_sample_rate instead of doing the more thorough check of has_tracing_enabled which will check either option.

As a workaround I seem to be able to just pass a dummy fixed sampling rate, since the TracesSampler always takes priority.

P.P.S. Looks like the workaround might not actually work, since I’m still seeing no profiles for traced transactions. Unless they take a while to process. But I will try some alternative sampling methods for the profiler before giving up.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 1
Comments: 28 (10 by maintainers)

Commits related to this issue

fix(profiling): Always use builtin time.sleep As pointed out in https://github.com/getsentry/sentry-python/issues/1813#issuecomment-1406636598, gevent patches the `time` module and `time.sleep` will ... — committed to getsentry/sentry-python by Zylphrex a year ago
fix(profiling): Always use builtin time.sleep (#1869) As pointed out in https://github.com/getsentry/sentry-python/issues/1813#issuecomment-1406636598, gevent patches the `time` module and `time.sle... — committed to getsentry/sentry-python by Zylphrex a year ago

Most upvoted comments

@Zylphrex Since the original issue has been resolved I will close the issue. I may open another one for greenlet tracking improvements.

I might take a look at the backend code myself to see what’s needed there to support tracking fibers in addition to threads.

Daverball on Mar 10, 2023

@Zylphrex Looks like delaying the start of the sampler thread does indeed do the trick.

Daverball on Feb 15, 2023

@Zylphrex I’ve made a little proof-of-concept for sampling all the currently active greenlets. It’s a little bit messy still, but it works. Cleaning it up properly will probably require some backend-support for an optional second identifier in addition to the thread_id for a fiber-like (i.e. in this case a greenlet) within a thread. Right now i’m just concatenating the two numbers into a string, if I add a hyphen or dot to separate the two ids, the Backend will reject the profile, if I try to generate a unique integer from the two integers instead of a string there also seem to be some problems on the backend, so right now there is some use of Union that is not very pretty for get_current_thread_id and related members/functions. https://github.com/Daverball/sentry-python/commit/e7d7dcd1a7392cc3a9f4fefbd65905a0b8e90499

Daverball on Feb 13, 2023

@Zylphrex Alright I think it might be related to uwsgi’s builtin static mapping and/or cache. If I enable them for the minimal repro case, then profiles stop working as well:

repro.ini

[uwsgi]
http = :9090
wsgi-file = repro.py
gevent = 100
enable-threads = True
cache2 = name=default,items=1000,purge_lru=True
static-map = /static=data/static
static-gzip-all = True
static-cache-paths = 60

Yes, If I remove the static mapping and cache from my application, then profiles start working there as well. I assume the way uwsgi forks the processes happens a bit differently with static maps.

Daverball on Feb 12, 2023

I managed to get one profile to show up as a complete fluke. The issue definitely is that the scheduler doesn’t work correctly with gevent though, so for pretty much any view it just does not get enough chances to record a sample. I will have to upgrade to 1.14 and retest, now that gevent is supported properly.

That being said I think the originally reported issue has been fixed and the workaround should no longer be necessary. But I will test with 1.14 (as soon as I have the time) before closing the issue.

Daverball on Jan 25, 2023

@sl0thentr0py I’m using my own framework (unfortunately I currently cannot share the source code), but it is WSGI and using the SentryWsgiMidleware along with a couple of custom hooks to extract information from the request and any SQL queries.

Daverball on Jan 3, 2023