sshkit: NoMethodError: undefined method `recv' for #

Intermittetently during deploys, the deploy will fail around 50% of the time with the following error - however executing cap deploy again (sometimes several attempts will be required) will usually produce a successful deploy.

Environment: Capistrano v3.2.1, SSHKit 1.4.0, net-ssh 2.9.1 under Ruby 2.1.1 deploying a Rails 4.0.2 application.

cap aborted!
NoMethodError: undefined method `recv' for #<IO:fd 47>
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/net-ssh-2.9.1/lib/net/ssh/buffered_io.rb:65:in `fill'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/net-ssh-2.9.1/lib/net/ssh/transport/packet_stream.rb:86:in `next_packet'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:178:in `block in poll_message'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:173:in `loop'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:173:in `poll_message'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:210:in `block in wait'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:208:in `loop'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:208:in `wait'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:87:in `initialize'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/net-ssh-2.9.1/lib/net/ssh.rb:202:in `new'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/net-ssh-2.9.1/lib/net/ssh.rb:202:in `start'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/sshkit-1.4.0/lib/sshkit/backends/connection_pool.rb:24:in `create_or_reuse_connection'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:173:in `ssh'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:126:in `block in _execute'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:123:in `tap'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:123:in `_execute'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:62:in `test'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/capistrano-3.2.1/lib/capistrano/tasks/deploy.rake:216:in `block (3 levels) in <top (required)>'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:54:in `instance_exec'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:54:in `run'
/Users/jason/.rvm/gems/ruby-2.1.1@avetars/gems/sshkit-1.4.0/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
Tasks: TOP => deploy:set_previous_revision
(See full trace by running task with --trace)
The deploy has failed with an error: #<NoMethodError: undefined method `recv' for #<IO:fd 47>>

About this issue

Original URL
State: closed
Created 10 years ago
Comments: 52 (18 by maintainers)

Most upvoted comments

OK, I’ve figured it out: defining methods with class << obj ... end is not thread safe in MRI!

Here’s a simple script to demonstrate:

require "thread"

def define_singleton_methods_in_thread
  Thread.new do
    100_000.times do
      obj = Object.new
      class << obj
        def one
        end

        def two
        end

        def three
        end
      end
      puts "FAIL" unless obj.respond_to?(:three)
    end
  end
end

# Test with 2 threads
2.times.map { define_singleton_methods_in_thread }.map(&:join)

On my machine this outputs:

FAIL
FAIL
FAIL

So I guess the only options here are:

Turn off threading (i.e. use :in => :sequence)
Guard Net::SSH.start with a mutex
Open a net-ssh PR to rewrite the code to avoid using singleton methods
Somehow get this fixed in MRI?

Thoughts on how to proceed, @leehambley ?

mattbrictson on May 13, 2015