rails: Puma raises WARNING: Detected 1 Thread(s) started in app boot with preload_app and workers
Steps to reproduce
- Uncomment the
preload_app
andworkers
commands. - Start server.
RAILS_ENV=production pumactl start
[20350] Puma starting in cluster mode...
[20350] * Version 3.12.1 (ruby 2.6.3-p62), codename: Llamas in Pajamas
[20350] * Min threads: 5, max threads: 5
[20350] * Environment: production
[20350] * Process workers: 2
[20350] * Preloading application
[20350] * Listening on tcp://0.0.0.0:3000
[20350] ! WARNING: Detected 1 Thread(s) started in app boot:
[20350] ! #<Thread:0x0000000005156350@/home/swistak/.rvm/gems/ruby-2.6.3/gems/activerecord-6.0.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:335 sleep> - /home/swistak/.rvm/gems/ruby-2.6.3/gems/activerecord-6.0.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep'
[20350] Use Ctrl-C to stop
[20350] - Worker 0 (pid: 20357) booted, phase: 0
[20350] - Worker 1 (pid: 20367) booted, phase: 0
Expected behavior
Lack of warning, unless it’s really dangerous. Is there a way to avoid that warning? If not possibly a documentation should mention that when you enable combination of those options (which config file recommends), you will get that warning
Actual behavior
No idea if this warning is dengerous or not, but probably should not happen with the default configuration.
It seems that connection pool is only creating a new thread in a production environment, does not happen in development.
System configuration
Rails version: 6.0.0pre
Ruby version: 2.6.3-p62
Puma version: 3.12.1
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 27
- Comments: 24 (21 by maintainers)
Commits related to this issue
- Mark Reaper thread as fork-safe Re: https://github.com/rails/rails/issues/37066#issuecomment-709403972 Puma (and some other multi-threaded application servers) may check the `Thread` list before and... — committed to nateberkopec/rails by nateberkopec 4 years ago
- Mark Reaper thread as fork-safe w/thread-local variable (#40399) Re: https://github.com/rails/rails/issues/37066#issuecomment-709403972 Puma (and some other multi-threaded application servers) may... — committed to rails/rails by nateberkopec 4 years ago
I’ve created https://github.com/rails/rails/pull/40399 as a resolution. Once the thread-local is decided, we’ll add to Puma.
For Puma, we plan a 5.1 release with this change somtime around the end of Nov/start of Dec. https://github.com/puma/puma/milestone/12
The warning is a bit…premature. It was added to puma on the assumption that the developer wouldn’t realize that the thread was also running in each fork. But there are cases where we legitimately want a background thread running in the main worker process. I’m not sure how to square those two things. Basically: There’s nothing inherently wrong with having threads running in your worker process, but it might not do what some people think. It’s a messaging problem.
@nateberkopec there’s an edge case where a mid-reap reaper may have the pool locked, and the post-fork process will itself attempt to
synchronize
the pool indiscard
… but that’s safe because a ruby mutex will unlock if its owning thread is MIA:The pool isn’t closed before fork; the pool is discarded (in the forked process) after fork.
Yup… and there’s one [or more, with multi-DB stuff involved] pool per forked process -> reaper[s] per forked process.
https://github.com/rails/rails/blob/b5b078cd99b0107fd8e31ff619578e5321182e0a/activerecord/lib/active_record/connection_adapters/pool_config.rb#L43-L47
Puma is warning that something could go wrong if the owner of the thread isn’t implemented correctly, but provides no way for us to indicate that yes, we do know what we’re doing. Alternative suggestions welcome, but right now I really don’t see a problem to solve here.
Oh! Thanks for the clarification. OK, it’s a false positive.
I misunderstood, and thought it was warning that the thread would NOT be running in each fork – I knew we wanted it to be. But in fact it IS running in each fork, and the puma warning is wrongly assuming it should not be.
OK, thanks @schneems , now this explanation is here for anyone else that finds it!
Does anyone have more info about this?
You still get this error under ActiveRecord 6.0.3.3 and puma 4.3.5, if you have puma configured with heroku’s recommendations. (multiple processes with preload, also
on_worker_boot { ActiveRecord::Base.establish_connection }
, don’t know if that’s relevant).puma is warning that a thread is started BEFORE fork happens, so may not exist where expected in forked process. It looks like the thread started is meant to be a “reaper” thread, eliminating abandoned connections from the connection pool.
It may not be running in all of your puma processes, meaning some of them may not be reaping abandoned connections. Maybe this isn’t a problem, but definitely isn’t as intended. Perhaps this should be re-opened?
Paging @schneems (hope you don’t mind, let me know if you do), since this happens under the recommended heroku puma configuration (which is where I noticed it), and heroku may want to take account.