rails: Test suite hangs after upgrading from 6.1 to 7.0

Steps to reproduce

We have a project with a multi-database (2 databases) configuration. When executing the test suite like follows the tests hang after some time.

PARALLEL_WORKERS=1 bin/rails test

As you can see in the sigdump all threads get stuck in load_interlock_aware_monitor.rb:17:in 'enter' I am currently trying to create an example project to reproduce this issue but maybe this info is enough for now to make some conclusions.

Sigdump at 2023-06-14 10:11:32 +0200 process 90089 (bin/rails)
  Thread #<Thread:0x00000001023db0f0 run> status=run priority=0
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `enter'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `block in mon_enter'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `mon_enter'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:22:in `block in synchronize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_adapters/abstract/query_cache.rb:91:in `clear_query_cache'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_handling.rb:343:in `block in clear_on_handler'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_handling.rb:342:in `each'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_handling.rb:342:in `clear_on_handler'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_handling.rb:272:in `clear_query_caches_for_current_thread'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `insert'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/persistence.rb:496:in `_insert_record'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/persistence.rb:1096:in `_create_record'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/counter_cache.rb:166:in `_create_record'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/locking/optimistic.rb:79:in `_create_record'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/attribute_methods/dirty.rb:222:in `_create_record'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/callbacks.rb:459:in `block in _create_record'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:107:in `run_callbacks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:929:in `_run_create_callbacks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/callbacks.rb:459:in `_create_record'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/timestamp.rb:108:in `_create_record'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/bullet-7.0.7/lib/bullet/active_record70.rb:6:in `_create_record'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/persistence.rb:1067:in `create_or_update'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/callbacks.rb:455:in `block in create_or_update'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/autosave_association.rb:370:in `around_save_collection_association'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:138:in `run_callbacks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/callbacks.rb:455:in `create_or_update'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/timestamp.rb:126:in `create_or_update'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/persistence.rb:648:in `save!'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/validations.rb:53:in `save!'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/transactions.rb:302:in `block in save!'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/transactions.rb:302:in `save!'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/suppressor.rb:54:in `save!'
      /Users/krebbl/Projects/ProCarement/care-center/app/models/medical_report.rb:121:in `save_report_as_document'
      /Users/krebbl/Projects/ProCarement/care-center/test/models/medical_report_test.rb:167:in `block (3 levels) in <class:MedicalReportTest>'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/testing/assertions.rb:34:in `assert_nothing_raised'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/testing/assertions.rb:250:in `_assert_nothing_raised_or_warn'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/testing/assertions.rb:102:in `assert_difference'
      /Users/krebbl/Projects/ProCarement/care-center/test/models/medical_report_test.rb:165:in `block (2 levels) in <class:MedicalReportTest>'
      /Users/krebbl/Projects/ProCarement/care-center/test/models/medical_report_test.rb:172:in `instance_exec'
      /Users/krebbl/Projects/ProCarement/care-center/test/models/medical_report_test.rb:172:in `block in create_test_from_should_hash'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/test.rb:102:in `block (3 levels) in run'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/test.rb:199:in `capture_exceptions'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/test.rb:97:in `block (2 levels) in run'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:296:in `time_it'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/test.rb:96:in `block in run'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:391:in `on_signal'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/test.rb:247:in `with_info_handler'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/test.rb:95:in `run'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-reporters-1.4.3/lib/minitest/reporters.rb:48:in `run_with_hooks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:1051:in `run_one_method'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:365:in `run_one_method'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:352:in `block (2 levels) in run'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:351:in `each'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:351:in `block in run'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:391:in `on_signal'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:378:in `with_info_handler'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:350:in `run'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/railties-7.0.5/lib/rails/test_unit/line_filtering.rb:10:in `run'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:182:in `block in __run'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:182:in `map'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:182:in `__run'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:159:in `run'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest.rb:83:in `block in autorun'
  Thread #<Thread:0x0000000109047888 /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb:40 sleep> status=sleep priority=0
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb:46:in `sleep'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb:46:in `block in spawn_thread'
  Thread #<Thread:0x0000000109443018@Timeout stdlib thread /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/timeout-0.3.2/lib/timeout.rb:101 sleep> status=sleep priority=0
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/timeout-0.3.2/lib/timeout.rb:113:in `sleep'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/timeout-0.3.2/lib/timeout.rb:113:in `wait'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/timeout-0.3.2/lib/timeout.rb:113:in `block (2 levels) in create_timeout_thread'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/timeout-0.3.2/lib/timeout.rb:111:in `synchronize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/timeout-0.3.2/lib/timeout.rb:111:in `block in create_timeout_thread'
  Thread #<Thread:0x000000010c94b918 /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:28 sleep_forever> status=sleep priority=0
      <internal:thread_sync>:18:in `pop'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:30:in `block (2 levels) in start'
  Thread #<Thread:0x000000010c94b800 /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:28 sleep_forever> status=sleep priority=0
      <internal:thread_sync>:18:in `pop'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:30:in `block (2 levels) in start'
  Thread #<Thread:0x000000010c94b710 /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:28 sleep_forever> status=sleep priority=0
      <internal:thread_sync>:18:in `pop'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:30:in `block (2 levels) in start'
  Thread #<Thread:0x000000010c94b620 /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:28 sleep_forever> status=sleep priority=0
      <internal:thread_sync>:18:in `pop'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:30:in `block (2 levels) in start'
  Thread #<Thread:0x000000010c94b530 /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:28 sleep_forever> status=sleep priority=0
      <internal:thread_sync>:18:in `pop'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:30:in `block (2 levels) in start'
  Thread #<Thread:0x000000010c94b440 /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:28 sleep_forever> status=sleep priority=0
      <internal:thread_sync>:18:in `pop'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:30:in `block (2 levels) in start'
  Thread #<Thread:0x000000010c94b350 /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:28 sleep_forever> status=sleep priority=0
      <internal:thread_sync>:18:in `pop'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:30:in `block (2 levels) in start'
  Thread #<Thread:0x000000010c94b260 /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:28 sleep_forever> status=sleep priority=0
      <internal:thread_sync>:18:in `pop'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/minitest-5.18.0/lib/minitest/parallel.rb:30:in `block (2 levels) in start'
  Thread #<Thread:0x000000010ce4f180@worker-1 /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:332 sleep_forever> status=sleep priority=0
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `enter'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `block in mon_enter'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `mon_enter'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:22:in `block in synchronize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_adapters/abstract/query_cache.rb:128:in `cache_sql'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_adapters/abstract/query_cache.rb:107:in `select_all'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/querying.rb:54:in `_query_by_sql'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:942:in `block in exec_main_query'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:962:in `skip_query_cache_if_necessary'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:928:in `exec_main_query'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:914:in `block in exec_queries'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:962:in `skip_query_cache_if_necessary'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:908:in `exec_queries'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:695:in `load'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:250:in `records'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/bullet-7.0.7/lib/bullet/active_record70.rb:46:in `records'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation/finder_methods.rb:520:in `find_take'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation/finder_methods.rb:98:in `take'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation/finder_methods.rb:473:in `find_one'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation/finder_methods.rb:457:in `find_with_ids'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation/finder_methods.rb:69:in `find'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/querying.rb:22:in `find'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/core.rb:271:in `find'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/globalid-1.1.0/lib/global_id/locator.rb:129:in `locate'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/globalid-1.1.0/lib/global_id/locator.rb:154:in `block in locate'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/globalid-1.1.0/lib/global_id/locator.rb:164:in `block in unscoped'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:881:in `_scoping'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:428:in `scoping'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/scoping/default.rb:43:in `unscoped'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/globalid-1.1.0/lib/global_id/locator.rb:164:in `unscoped'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/globalid-1.1.0/lib/global_id/locator.rb:154:in `locate'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/globalid-1.1.0/lib/global_id/locator.rb:16:in `locate'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/arguments.rb:148:in `deserialize_global_id'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/arguments.rb:132:in `deserialize_argument'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/arguments.rb:43:in `block in deserialize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/arguments.rb:43:in `map'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/arguments.rb:43:in `deserialize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/core.rb:190:in `deserialize_arguments'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/core.rb:180:in `deserialize_arguments_if_needed'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/execution.rb:44:in `perform_now'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/instrumentation.rb:14:in `block in perform_now'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/instrumentation.rb:25:in `block in instrument'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/notifications.rb:206:in `block in instrument'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/notifications.rb:206:in `instrument'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/instrumentation.rb:35:in `instrument'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/instrumentation.rb:14:in `perform_now'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/logging.rb:18:in `block in perform_now'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/tagged_logging.rb:99:in `block in tagged'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/tagged_logging.rb:37:in `tagged'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/tagged_logging.rb:99:in `tagged'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/logging.rb:25:in `tag_logger'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/logging.rb:18:in `perform_now'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/execution.rb:24:in `block in execute'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/railtie.rb:54:in `block (4 levels) in <class:Railtie>'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/execution_wrapper.rb:92:in `wrap'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/reloader.rb:72:in `block in wrap'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/execution_wrapper.rb:92:in `wrap'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/reloader.rb:71:in `wrap'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/railtie.rb:53:in `block (3 levels) in <class:Railtie>'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:127:in `instance_exec'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:138:in `run_callbacks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/execution.rb:22:in `execute'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/queue_adapters/async_adapter.rb:70:in `perform'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
  Thread #<Thread:0x000000010d989d20@worker-2 /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:332 sleep_forever> status=sleep priority=0
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `enter'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `block in mon_enter'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `mon_enter'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:22:in `block in synchronize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_adapters/abstract/query_cache.rb:128:in `cache_sql'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/connection_adapters/abstract/query_cache.rb:107:in `select_all'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/querying.rb:54:in `_query_by_sql'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:942:in `block in exec_main_query'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:962:in `skip_query_cache_if_necessary'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:928:in `exec_main_query'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:914:in `block in exec_queries'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:962:in `skip_query_cache_if_necessary'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:908:in `exec_queries'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:695:in `load'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:250:in `records'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/bullet-7.0.7/lib/bullet/active_record70.rb:46:in `records'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation/finder_methods.rb:520:in `find_take'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation/finder_methods.rb:98:in `take'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation/finder_methods.rb:473:in `find_one'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation/finder_methods.rb:457:in `find_with_ids'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation/finder_methods.rb:69:in `find'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/querying.rb:22:in `find'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/core.rb:271:in `find'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/globalid-1.1.0/lib/global_id/locator.rb:129:in `locate'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/globalid-1.1.0/lib/global_id/locator.rb:154:in `block in locate'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/globalid-1.1.0/lib/global_id/locator.rb:164:in `block in unscoped'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:881:in `_scoping'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/relation.rb:428:in `scoping'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activerecord-7.0.5/lib/active_record/scoping/default.rb:43:in `unscoped'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/globalid-1.1.0/lib/global_id/locator.rb:164:in `unscoped'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/globalid-1.1.0/lib/global_id/locator.rb:154:in `locate'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/globalid-1.1.0/lib/global_id/locator.rb:16:in `locate'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/arguments.rb:148:in `deserialize_global_id'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/arguments.rb:132:in `deserialize_argument'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/arguments.rb:43:in `block in deserialize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/arguments.rb:43:in `map'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/arguments.rb:43:in `deserialize'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/core.rb:190:in `deserialize_arguments'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/core.rb:180:in `deserialize_arguments_if_needed'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/execution.rb:44:in `perform_now'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/instrumentation.rb:14:in `block in perform_now'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/instrumentation.rb:25:in `block in instrument'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/notifications.rb:206:in `block in instrument'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/notifications.rb:206:in `instrument'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/instrumentation.rb:35:in `instrument'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/instrumentation.rb:14:in `perform_now'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/logging.rb:18:in `block in perform_now'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/tagged_logging.rb:99:in `block in tagged'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/tagged_logging.rb:37:in `tagged'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/tagged_logging.rb:99:in `tagged'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/logging.rb:25:in `tag_logger'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/logging.rb:18:in `perform_now'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/execution.rb:24:in `block in execute'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/railtie.rb:54:in `block (4 levels) in <class:Railtie>'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/execution_wrapper.rb:92:in `wrap'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/reloader.rb:72:in `block in wrap'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/execution_wrapper.rb:92:in `wrap'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/reloader.rb:71:in `wrap'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/railtie.rb:53:in `block (3 levels) in <class:Railtie>'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:127:in `instance_exec'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activesupport-7.0.5/lib/active_support/callbacks.rb:138:in `run_callbacks'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/execution.rb:22:in `execute'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/activejob-7.0.5/lib/active_job/queue_adapters/async_adapter.rb:70:in `perform'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
      /Users/krebbl/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'

Expected behavior

The tests should just run through

Actual behavior

The tests get stuck after some while

System configuration

Rails version: 7.0.5 Ruby version: 3.2.0

About this issue

Original URL
State: open
Created a year ago
Comments: 21 (9 by maintainers)

Most upvoted comments

Thanks @nbelzer! I think you found the root of the problems.

I had the same problems for 2 different Rails apps after updating to 7.1

Setting the queue_adapter to :inline led to a different error during the tests:

NotImplementedError: Use a queueing backend to enqueue jobs in the future. Read more at https://guides.rubyonrails.org/active_job_basics.html

But doing what @nbelzer did solved the problem for me:

# environments/test.rb
config.active_job.queue_adapter = :test

I think this line should be automatically added to environments/test.rb while upgrading Rails to 7.1. Unless there is a good reason (like test performance?) why we would want :async as the default queue_adapter, but then this issue needs to be fixed.

edit:

btw. in my case I experienced the problems with minitest. So it doesn’t affect only rspec.

arusa on Oct 25, 2023

I had a similar issue when upgrading from 7.0.8 to 7.1.1. Spent quite some hours trying to figure out why it hangs but have not been succesful. What I’ve found so far:

The default queue_adapter on the test environment is set to :async unless we include the ActiveJob::TestHelper or explicitly set the queue_adapter to :test.
Having the async queue adapter allows jobs to run outside of the transaction that wraps the tests. This can cause an ActiveJob::DeserializationError as the original object passed to the job no longer exists in the database.
This is exactly what happens whenever my test suite hangs, a job is executed, but fails due to being executed after the transaction is rolled back.
The job running outside of the transaction and failing due to the record passed along no longer existing already existed in Rails 7.0.8 (I rolled back the application locally to test this), but the test suite would not hang on it.

My assumption then is that somehow the thread running the job asynchronously does not exit properly when the job errors and therefore blocks further test execution. However, I’m not sure how to diagnose this.

I was able to recreate the ‘job executing outside of the test transaction behaviour’ in a new rails project for which I created a repo: nbelzer/rails-async-test-queue-example. However, I was not able to replicate the test suite hanging in this repo, which would indicate that I’m still missing an ingredient.

Regardless in all situations where the test suite hangs for me, a failed job with a ActiveJob::DeserializationError is the last part of the test.log. This leads me to believe that the two are related and fixing the first, would solve the latter.

The solution for me has been to set the queue_adapter to :test in the test environment, but I wonder why the default is :async.

I’ll keep an eye on this issue as I’d like to figure out what is causing the hanging. I’m out of clues for now, but hopefully, by sharing these findings someone else is able to find the next one.

nbelzer on Oct 19, 2023

Strangely enough, I had this issue when upgrading from 7.0 to 7.1 version. I fixed it by setting queue_adapter option like @rafaelfranca mentioned here.

# config/environments/test.rb

Rails.application.configure do
  config.active_job.queue_adapter = :inline
end

rainerborene on Oct 6, 2023

Could you check if changing the adapter to be the inline your issue goes away?

Rails.application.config.active_job.queue_adapter = :inline

rafaelfranca on Jun 26, 2023