excon: Intermittent "broken pipe" errors

I’m using Carrierwave 1.1.0, Fog 1.41.0, Excon 0.58.0 to upload files to Rackspace from a Ruby on Rails server. I’ve been seeing “Broken Pipe” errors happening intermittently at the point the code tries these file uploads. I see it a few times a week, but it isn’t something I’ve been able to reproduce locally, because it happens relatively rarely.

The only information I’ve seen about this type of error online is that it often happens because people aren’t pointing to the correct region when trying to upload files, but the region in my config (IAD) matches what the Rackspace site tells me to use.

My Carrierwave config (minus username, API key, fog_directory) looks like this:

CarrierWave.configure do |config|
  config.fog_provider = "fog/rackspace/storage" 
  config.fog_credentials = {
      provider: 'Rackspace',
      rackspace_username: 'XXXX',
      rackspace_api_key: 'XXXX',
      rackspace_region: :iad
  }
  config.fog_directory = 'XXXX'
end

The error messages I get look like this: !ruby/exception:Excon::Error::Socket message: Broken pipe (Errno::EPIPE) socket_error: !ruby/exception:Errno::EPIPE message: Broken pipe

I turned on Excon debug logging, and it looks like in the error case Excon tries the request several times and then shows the broken pipe message. The total time from request start to error message is less than 2 seconds, so I doubt this is a timeout issue with Rackspace, but I don’t see any info in the Excon output or other logs that explain why it’s retrying or failing.

I’ve tried a few different versions of Carrierwave and Fog and haven’t found a configuration that solves this. Is there a way I can get more information about what’s going on at the point the Excon request to Rackspace happens, or have other people run into this?

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 39 (11 by maintainers)

Commits related to this issue

Rescue Excon::Error::Socket and authenticate Rackspace users started reporting intermittent "broken pipe" errors that seem to be related to authentication. This rescues those errors and attempts to a... — committed to codeodor/fog-rackspace by codeodor 7 years ago

Most upvoted comments

We have never ran into the issue again after applying the fix.

codeodor on Jul 26, 2018

I’m going to go ahead and close, as in rereading (if I’m not mistaken) it is really a rackspace/fog-rackspace issue more than an excon one per se? I’d still be happy to support a patch on fog-rackspace if anyone wants to work on that though. Thanks!

geemus on Jul 26, 2018

@codeodor understandable. Yeah, I think it should be a pretty safe change (probably not going to make anything worse anyway). Would you be able to do the honors of making a PR for us? Thanks!

geemus on Oct 3, 2017

I made a change to retry the upload on broken pipe errors on Sept 12 and since then have only seen the broken pipe error happen twice, but the new change’s retry succeeded both times. My monkey patch looks like this (I put this in a file at config/fog_overrides.rb):

module Fog
  module Rackspace
    class Service
      def request(params, parse_json = true)
        first_attempt = true
        begin
          response = @connection.request(request_params(params))
        rescue Excon::Errors::Unauthorized => error
          raise error unless first_attempt
          first_attempt = false
          authenticate
          retry
        rescue Excon::Error::Socket => ees
          if ees.message && ees.message.downcase.include?('broken pipe')
            raise ees unless first_attempt
            first_attempt = false
            authenticate
            retry
          end
        end

        process_response(response) if parse_json
        response
      end
    end
  end
end

paulvwarner on Sep 22, 2017

Given what you both describe, I could certainly imagine a bad token being the issue. In particular, I wouldn’t expect other socket issues to be persistent. If you reset the socket and retry, the retry should theoretically work. That it continues to fail makes it seem more likely it is something that remains constant (and token sounds like a good candidate).

I don’t know the context by which it was added (it may well have been something like this), but it appears fog-rackspace has an rackspace_must_reauthenticate option, that if you set it to true, it will always get a new token. I suspect this will cause performance impact, but if that is acceptable it might work more consistently.

https://github.com/fog/fog-rackspace/blob/master/lib/fog/rackspace/storage.rb#L57

geemus on Aug 31, 2017

For me, it’s happening for image uploads where I’m constraining the dimensions of the image it’s possible to upload, so the file size doesn’t get too crazy - it’s usually between 300 and 800 KB.

I was wondering if Excon is getting a response when it’s doing these retries - I don’t see it in the output. (I attached an example of the Excon output for a failure, with the request body and auth info removed.) If there’s a response coming from Rackspace in these cases, I can probably take that to Rackspace and see if they have more info.

excon request.txt

paulvwarner on Aug 29, 2017