request: Request with interval() eventually produces ESOCKETTIMEDOUT and ECONNRESET

I have a persistent process that goes off and crawls urls every X minutes.

node v0.10.33
request 2.44.0
ubuntu 14.04 on EC2
uname -a
Linux ip-myip 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

I get into this state periodically where all my requests get ESOCKETTIMEDOUT and ECONNRESET errors. Also when it gets into this state, my tcp connections for node process is stuck at only 13 established (these are my mongodb connections):

lsof -p 9514 | grep TCP | wc -l
13

If I restart node things go back to normal (for a while). The lsof command goes up and down as expected, until issue is hit again.

This is the block of code that is called via interval()

var separateReqPool = {maxSockets: 100},
      reqObj = {
            pool: separateReqPool,
            uri: feed.feedUrl,
            headers: {"User-Agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2"},
            timeout: 5000,
            jar: false,
            maxRedirects: 5,
            strictSSL: false
      };

    request(reqObj, function(error){
        if(error){
            cfg.logger.error("ERR: bad request: "+error, error);
        }
    })
        .on('response', function(response){
            ...
        })
        .on('error', function(error){
            cfg.logger.error("ERR: "+error, error);
        })
        .on('end', function(){
            cfg.logger.info("feed pull complete - "+feed.feedUrl);            
        });

Any ideas? thanks in advance.

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Comments: 15 (4 by maintainers)

Commits related to this issue

Most upvoted comments

@doapp-ryanp Sorry for bumping, but just to make it clear as I’m having this problem too - Are you passing agent:false in Request’s options?

As in -

new Request({agent:false}, (error, response, body) => {
         ....
        });

Or is it something internal?

Why close this entire issue because of a bad url in a comment? This is def an issue that others have run into enough to completely use a different library.

It is hard to consistently reproduce, which is why I have not posted a script+httpbin.org test case. The inconsistency is also why this bug is so frustrating.

I personally have switched to https://github.com/substack/hyperquest, but I think leaving this issue open for more investigation is a benefit for the entire request community.