mechanize: Net::HTTP::Persistent::Error: too many connection resets (due to Connection reset by peer - Errno::ECONNRESET) after 2 requests on 14759220
I am trying to do some simple screen scraping on etrade’s website but am getting a similar issue that was reported by someone earlier. I read through that message thread but it doesn’t look like it was ever resolved.
Here is the error I am getting back:
Net::HTTP::Persistent::Error: too many connection resets (due to Connection reset by peer - Errno::ECONNRESET) after 2 requests on 14759220
from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/net-http-persistent-1.8/lib/net/http/persistent.rb:446:in `rescue in request'
from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/net-http-persistent-1.8/lib/net/http/persistent.rb:422:in `request'
from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/mechanize-2.0.1/lib/mechanize/http/agent.rb:204:in `fetch'
from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/mechanize-2.0.1/lib/mechanize.rb:628:in `post_form'
from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/mechanize-2.0.1/lib/mechanize.rb:520:in `submit'
from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/mechanize-2.0.1/lib/mechanize/form.rb:167:in `submit'
from (irb):74
from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/railties-3.0.9/lib/rails/commands/console.rb:44:in `start'
from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/railties-3.0.9/lib/rails/commands/console.rb:8:in `start'
from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/railties-3.0.9/lib/rails/commands.rb:23:in `<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'
Here is my code:
require 'rubygems'
require 'mechanize'
agent = Mechanize.new
login_page = agent.get("https://www.etrade.com")
form = login_page.form_with(:action => '/login.fcc')
form.USER = "test"
form.PASSWORD = "test12"
form.submit
Any ideas?
Thanks
About this issue
- Original URL
- State: closed
- Created 15 years ago
- Reactions: 1
- Comments: 104 (38 by maintainers)
Commits related to this issue
- Use the latest net-http-persistent with idle timeouts. Issue #123 — committed to sparklemotion/mechanize by drbrain 15 years ago
- Add Mechanize#retry_change_requests to allow users to say "POST is OK!". Issue #123 — committed to sparklemotion/mechanize by drbrain 15 years ago
- Set idle_timeout after the HTTP instance has been created. Issue #123 — committed to sparklemotion/mechanize by drbrain 15 years ago
- Allow disabling of keep-alive requests again via an update to net-http-persistent. Issue #123 — committed to sparklemotion/mechanize by drbrain 15 years ago
- Collect passages without tabs and newlines — committed to wfth/collector by isaacjwilliams 9 years ago
- Update Mechanize agent to TLSv1.2 See https://github.com/sparklemotion/mechanize/issues/123 — committed to snehankekre/the_hills1 by deleted user 8 years ago
Here is what looks to be going on:
I see this error when I run a request, then after a brief pause, run another request, using SSL. It does not happen when I run the requests back to back. I got a full backtrace from line 460 of persistent.rb, and that showed that the IOError was being raised from buffering.rb:145 in read_nonblock at a call to sysread_nonblock, called from net/protocol.rb:135 in rbuf_fill.
Digging down, it looks like sysread_nonblock is implemented (in 1.9.2) in the core file ossl_ssl.c, and only returns an IOError if one of two errors occurred, SSL_ERROR_ZERO_RETURN or SSL_ERROR_SYSCALL. The documentation here: http://www.openssl.org/docs/ssl/SSL_get_error.html indicates that either the connection has been closed, or an EOF was read that violates the SSL protocol. I’m guessing it is the first case because of the behavior with delays. So it looks like the problem is that regardless of the Connection: Keep-Alive setting, the SSL connection can go down and you get an EOFError. In the non-SSL situation, there is a similar problem with IO.read_nonblock() also having the possibility of returning EOFError.
The problem is going to be figuring out when that is the case so you can retry safely for non-idempotent queries. knu’s workaround will work if you know it is safe to do so. You can’t just always retry on EOFError because other levels can throw EOFError for different reasons.
Certs, CA paths, ssl_version, keep_alive, idle_timeout, retry_change_requests, etc…none of the suggestions above have had any effect at eliminating this bug from intermittently occurring in our high-volume production scrapers based on Mechanize.
As an experiment, I created a monkey-patch that simply shuts down the underlying persistent connection and tries again with a new one, whenever this error is caught. I’ve now been using this in production for high-volume scrapers for a few months. It has 100% eliminated this problem with no observable negative side-effects. YMMV.
Here’s my post about this workaround, along with the code:
http://scottwb.com/blog/2013/11/09/defeating-the-infamous-mechanize-too-many-connection-resets-bug/