delayed_job: IOError: closed stream error when starting worker process in Rails 6.1 on Ubuntu
Steps to reproduce:
- Use a machine running Ubuntu 20.04 with Ruby (I used 2.6.5 but I don’t think it matters) and Bundler
- Create a new Rails 6.1 application
- Add
gem 'daemons'andgem 'delayed_job_active_record'to Gemfile bundle install- Generate the DJ table with
bundle exec rails generate delayed_job:active_recordthenbundle exec rake db:migrate - Start a worker process using
bundle exec ruby bin/delayed_job start - The worker starts but sometimes (not always, I think), the following error is reported in the console:
$ bundle exec ruby bin/delayed_job start
E, [2021-01-04T16:05:25.312502 #203830] ERROR -- : Exception rescued in listen-run_thread:
IOError: closed stream
/home/tim/.rvm/gems/ruby-2.6.5/gems/rb-inotify-0.10.1/lib/rb-inotify/notifier.rb:317:in `select'
/home/tim/.rvm/gems/ruby-2.6.5/gems/rb-inotify-0.10.1/lib/rb-inotify/notifier.rb:317:in `readpartial'
/home/tim/.rvm/gems/ruby-2.6.5/gems/rb-inotify-0.10.1/lib/rb-inotify/notifier.rb:290:in `read_events'
/home/tim/.rvm/gems/ruby-2.6.5/gems/rb-inotify-0.10.1/lib/rb-inotify/notifier.rb:263:in `process'
/home/tim/.rvm/gems/ruby-2.6.5/gems/rb-inotify-0.10.1/lib/rb-inotify/notifier.rb:236:in `block in run'
/home/tim/.rvm/gems/ruby-2.6.5/gems/rb-inotify-0.10.1/lib/rb-inotify/notifier.rb:232:in `synchronize'
/home/tim/.rvm/gems/ruby-2.6.5/gems/rb-inotify-0.10.1/lib/rb-inotify/notifier.rb:232:in `run'
/home/tim/.rvm/gems/ruby-2.6.5/gems/listen-3.4.0/lib/listen/adapter/linux.rb:41:in `_run'
/home/tim/.rvm/gems/ruby-2.6.5/gems/listen-3.4.0/lib/listen/adapter/base.rb:79:in `block in start'
/home/tim/.rvm/gems/ruby-2.6.5/gems/listen-3.4.0/lib/listen/thread.rb:26:in `rescue_and_log'
/home/tim/.rvm/gems/ruby-2.6.5/gems/listen-3.4.0/lib/listen/thread.rb:18:in `block in new'
--- Thread.new ---
/home/tim/.rvm/gems/ruby-2.6.5/gems/listen-3.4.0/lib/listen/adapter/base.rb:75:in `start'
/usr/share/rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/forwardable.rb:230:in `start'
/home/tim/.rvm/gems/ruby-2.6.5/gems/listen-3.4.0/lib/listen/listener.rb:71:in `block in <class:Listener>'
/home/tim/.rvm/gems/ruby-2.6.5/gems/listen-3.4.0/lib/listen/fsm.rb:121:in `instance_eval'
/home/tim/.rvm/gems/ruby-2.6.5/gems/listen-3.4.0/lib/listen/fsm.rb:121:in `call'
/home/tim/.rvm/gems/ruby-2.6.5/gems/listen-3.4.0/lib/listen/fsm.rb:102:in `transition_with_callbacks!'
/home/tim/.rvm/gems/ruby-2.6.5/gems/listen-3.4.0/lib/listen/fsm.rb:69:in `transition'
/home/tim/.rvm/gems/ruby-2.6.5/gems/listen-3.4.0/lib/listen/listener.rb:92:in `start'
/home/tim/.rvm/gems/ruby-2.6.5/gems/activesupport-6.1.0/lib/active_support/evented_file_update_checker.rb:109:in `start'
/home/tim/.rvm/gems/ruby-2.6.5/gems/activesupport-6.1.0/lib/active_support/evented_file_update_checker.rb:89:in `block in initialize'
/home/tim/.rvm/gems/ruby-2.6.5/gems/activesupport-6.1.0/lib/active_support/fork_tracker.rb:36:in `each'
/home/tim/.rvm/gems/ruby-2.6.5/gems/activesupport-6.1.0/lib/active_support/fork_tracker.rb:36:in `check!'
/home/tim/.rvm/gems/ruby-2.6.5/gems/activesupport-6.1.0/lib/active_support/fork_tracker.rb:14:in `fork'
/home/tim/.rvm/gems/ruby-2.6.5/gems/activesupport-6.1.0/lib/active_support/fork_tracker.rb:26:in `fork'
/home/tim/.rvm/gems/ruby-2.6.5/gems/daemons-1.3.1/lib/daemons/daemonize.rb:10:in `safefork'
/home/tim/.rvm/gems/ruby-2.6.5/gems/daemons-1.3.1/lib/daemons/daemonize.rb:67:in `call_as_daemon'
/home/tim/.rvm/gems/ruby-2.6.5/gems/daemons-1.3.1/lib/daemons/application.rb:279:in `start_proc'
/home/tim/.rvm/gems/ruby-2.6.5/gems/daemons-1.3.1/lib/daemons/application.rb:305:in `start'
/home/tim/.rvm/gems/ruby-2.6.5/gems/daemons-1.3.1/lib/daemons/controller.rb:56:in `run'
/home/tim/.rvm/gems/ruby-2.6.5/gems/daemons-1.3.1/lib/daemons.rb:199:in `block in run_proc'
/home/tim/.rvm/gems/ruby-2.6.5/gems/daemons-1.3.1/lib/daemons/cmdline.rb:121:in `catch_exceptions'
/home/tim/.rvm/gems/ruby-2.6.5/gems/daemons-1.3.1/lib/daemons.rb:198:in `run_proc'
/home/tim/.rvm/gems/ruby-2.6.5/gems/delayed_job-4.1.9/lib/delayed/command.rb:123:in `run_process'
/home/tim/.rvm/gems/ruby-2.6.5/gems/delayed_job-4.1.9/lib/delayed/command.rb:104:in `block in daemonize'
/home/tim/.rvm/gems/ruby-2.6.5/gems/delayed_job-4.1.9/lib/delayed/command.rb:102:in `times'
/home/tim/.rvm/gems/ruby-2.6.5/gems/delayed_job-4.1.9/lib/delayed/command.rb:102:in `daemonize'
bin/delayed_job:5:in `<main>'
delayed_job: process with pid 203830 started.
This does not happen with DJ 4.1.9 in Rails 6.0.3.4. I’ve tried it on macOS and the error does not appear. I’ve also tried changing the Ruby version to 3.0.0 and the error still appears.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 11
- Comments: 15 (1 by maintainers)
For anyone still encountering this issue, I was able to identify the root issue with inspiration from this stackoverflow answer
When running the delayed job process in daemonized mode
bundle exec ruby bin/delayed_job startFrom current script generator:
I too a look into the daemonize gem and found that when starting up, it actually CLOSES ALL IO STREAMS!!. Modifying global state is always fun 👍 .
Within the script file, it first boots Rails via loading
config/environmentand especially with the config optioneager_load = truemeans that if there is any code during the bootup process that opens up IOs (e.g. I was opening up cache connections via the Dalli gem) will be closed after the call todaemonize.My current workaround is to ensure that Rails is loaded AFTER
daemonizeThis explains also why running it via rake
rake jobs:workdoes not have any issues as it does not daemonize the process.Same issue here
Same issue here. Happens in both development and production. Using the rake task (
rake jobs:work) seems to work ok. But that’s not ideal for production mode@allenjgreer Your workaround solved it here on macOS and FreeBSD. Thank you very much!
Same issue with Rails 6, all versions tried so far, after upgrading from 5. Possibly related to this issue: https://github.com/collectiveidea/delayed_job/issues/1099