newrelic-ruby-agent: Delayed job deadlocks in development

Description

When running rake jobs:work in development mode with a newrelic-enabled app, the main thread & the newrelic sampling thread eventually deadlock each other. No more jobs get run, and the process becomes unable to exit on ctrl-c.

Expected Behavior

No deadlocks! They’re not usually expected.

Troubleshooting or NR Diag results

Provide any other relevant log data.

Here’s the main thread following a deadlock:

lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `enter'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `block in mon_enter'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `mon_enter'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:22:in `block in synchronize'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:597:in `checkin'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:551:in `block (3 levels) in clear_reloadable_connections'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:548:in `each'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:548:in `block (2 levels) in clear_reloadable_connections'
lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:547:in `block in clear_reloadable_connections'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:749:in `block in with_exclusively_acquired_all_connections'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:824:in `with_new_connections_blocked'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:747:in `with_exclusively_acquired_all_connections'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:546:in `clear_reloadable_connections'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:570:in `clear_reloadable_connections!'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:1093:in `each'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:1093:in `clear_reloadable_connections!'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_handling.rb:351:in `clear_reloadable_connections!'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/railtie.rb:239:in `block (3 levels) in <class:Railtie>'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:427:in `instance_exec'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:427:in `block in make_lambda'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:603:in `catch'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:603:in `block in default_terminator'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:199:in `block in halting'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:512:in `block in invoke_before'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:512:in `each'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:512:in `invoke_before'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:105:in `run_callbacks'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/reloader.rb:120:in `class_unload!'
lib/ruby/gems/2.7.0/gems/railties-6.1.4/lib/rails/application/finisher.rb:244:in `block (2 levels) in <module:Finisher>'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:427:in `instance_exec'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:427:in `block in make_lambda'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:603:in `catch'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:603:in `block in default_terminator'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:199:in `block in halting'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:512:in `block in invoke_before'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:512:in `each'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:512:in `invoke_before'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/callbacks.rb:105:in `run_callbacks'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/execution_wrapper.rb:111:in `run!'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/reloader.rb:114:in `run!'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/reloader.rb:53:in `block (2 levels) in reload!'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/reloader.rb:52:in `tap'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/reloader.rb:52:in `block in reload!'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/reloader.rb:51:in `reload!'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:330:in `reload!'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:187:in `block (2 levels) in start'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:172:in `loop'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:172:in `block in start'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/tagged_logging.rb:99:in `block in tagged'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/tagged_logging.rb:37:in `tagged'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/tagged_logging.rb:99:in `tagged'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:61:in `block in initialize'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:79:in `block in add'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:79:in `block in add'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:66:in `execute'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:40:in `run_callbacks'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:171:in `start'
lib/ruby/gems/2.7.0/gems/delayed_job-4.1.9/lib/delayed/tasks.rb:9:in `block (2 levels) in <main>'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `block in execute'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `each'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `execute'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:188:in `invoke'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:160:in `invoke_task'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `each'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block in top_level'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:125:in `run_with_threads'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:110:in `top_level'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:83:in `block in run'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:186:in `standard_exception_handling'
lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:80:in `run'
bin/rake:6:in `<main>'

and NewRelic’s agent thread:

lib/ruby/2.7.0/monitor.rb:108:in `sleep'
lib/ruby/2.7.0/monitor.rb:108:in `wait'
lib/ruby/2.7.0/monitor.rb:108:in `wait_for_cond'
lib/ruby/2.7.0/monitor.rb:108:in `wait'
lib/ruby/2.7.0/monitor.rb:116:in `wait_while'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/share_lock.rb:220:in `wait_for'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/share_lock.rb:190:in `block in yield_shares'
lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/share_lock.rb:189:in `yield_shares'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:205:in `block in execute'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:204:in `execute'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/mysql/database_statements.rb:50:in `execute'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:215:in `execute_and_free'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/mysql/database_statements.rb:55:in `exec_query'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:532:in `select'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:69:in `select_all'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/abstract/query_cache.rb:103:in `select_all'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/connection_adapters/mysql/database_statements.rb:12:in `select_all'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/relation/calculations.rb:308:in `block in execute_simple_calculation'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/relation.rb:861:in `skip_query_cache_if_necessary'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/relation/calculations.rb:308:in `execute_simple_calculation'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/relation/calculations.rb:269:in `perform_calculation'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/relation/calculations.rb:144:in `calculate'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/instrumentation/active_record_prepend.rb:85:in `block in calculate'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent.rb:681:in `with_database_metric_name'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/instrumentation/active_record_prepend.rb:84:in `calculate'
lib/ruby/gems/2.7.0/gems/activerecord-6.1.4/lib/active_record/relation/calculations.rb:51:in `count'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/samplers/delayed_job_sampler.rb:56:in `count'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/logger_silence.rb:18:in `block in silence'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/logger_thread_safe_level.rb:52:in `log_at'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/logger_silence.rb:18:in `silence'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/logger.rb:64:in `block (3 levels) in broadcast'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/logger_silence.rb:18:in `block in silence'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/logger_thread_safe_level.rb:52:in `log_at'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/logger_silence.rb:18:in `silence'
lib/ruby/gems/2.7.0/gems/activesupport-6.1.4/lib/active_support/logger.rb:62:in `block (2 levels) in broadcast'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/samplers/delayed_job_sampler.rb:45:in `failed_jobs'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/samplers/delayed_job_sampler.rb:65:in `poll'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/sampler_collection.rb:41:in `block in poll_samplers'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/sampler_collection.rb:39:in `delete_if'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/sampler_collection.rb:39:in `poll_samplers'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/sampler_collection.rb:13:in `block in initialize'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/event_listener.rb:40:in `block in notify'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/event_listener.rb:38:in `each'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/event_listener.rb:38:in `notify'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/agent.rb:1143:in `transmit_data'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/agent.rb:619:in `block in create_and_run_event_loop'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/event_loop.rb:148:in `block in dispatch_event'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/event_loop.rb:146:in `each'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/event_loop.rb:146:in `dispatch_event'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/event_loop.rb:111:in `run_once'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/event_loop.rb:99:in `run'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/agent.rb:641:in `create_and_run_event_loop'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/agent.rb:701:in `block (2 levels) in deferred_work!'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent.rb:501:in `disable_all_tracing'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/agent.rb:698:in `block in deferred_work!'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/agent.rb:678:in `catch_errors'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/agent.rb:697:in `deferred_work!'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/agent.rb:724:in `block in start_worker_thread'
lib/ruby/gems/2.7.0/gems/newrelic_rpm-7.1.0/lib/new_relic/agent/threading/agent_thread.rb:14:in `block in create'

Steps to Reproduce

I found this helpful for narrowing down the problem:

NewRelic::Agent::Samplers::DelayedJobSampler.prepend(
  Module.new {
    def count(query)
      puts "#{Time.now} Fetching count #{query}"
      t = nil
      Rails.logger.silence do
        t = super
      end
      puts "#{Time.now} Count: #{t}"
      t
    rescue Exception => e
      puts e.inspect
      raise
    end
  }
)

NewRelic::Agent::EventLoop.prepend(
  Module.new {
    def fire_every(interval, event)
      interval = interval / 10
      puts "Firing event #{event} every #{interval} seconds."
      super(interval, event)
    end
  }
)

which logs before and after the DJ sampling occurs, and makes the sampling more frequent so you don’t have to wait hours for it to occur. When the deadlock occurs you’ll see something like

2021-07-01 11:55:30 +0100 Fetching count failed_at is not NULL
2021-07-01 11:55:30 +0100 Count: 4
2021-07-01 11:55:30 +0100 Fetching count locked_by is not NULL

in the log, followed by no other output.

Your Environment

Newrelic agent 7.1.0 Ruby 2.7.3 Rails 6.1.4 macOS 11.4

Additional context

I wonder if the DJ sampling needs wrapping in Rails.application.executor, but not totally sure.