got: Request hangs up with GOT v11

Describe the bug

Node.js version: 12.16.3
OS & version: Debian Buster

Actual behavior

Request gets stuck while downloading. …

Expected behavior

Request should either timeout or give correct response …

Code to reproduce


const agent = {
  http: new http.Agent({ keepAlive: true }),
  https: new https.Agent({ keepAlive: true })
};

var requestOptions = {
    timeout: 45000, // milliseconds
    responseType: 'buffer',
    retry: 0,
    maxRedirects: 5,
    dnsCache: false,
    headers: {
      'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0) Gecko/20100101 Firefox/57.0',
      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
      'Accept-Language': 'en-US,en;q=0.5'
    },
    shared: false,
    agent,
    cache: cacheStore
  };

got.get("https://www.empiresuppliesonline.co.uk/ekmps/shops/empiresupplies/images/Waring-X-Prep-Kitchen-Blender-2523-p.jpg", requestOptions);

I am using https://www.npmjs.com/package/improved-cacheable-request as cache store.

Please note I download around 500 such URLs per minute but the issue majorly happens with this given domain name.

Checklist

I have read the documentation.
I have tried my code with the latest version of Node.js and Got.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 19

Most upvoted comments

I think this might the issue we’re currently having in our project.

Minimal repo (Node v12.18.0) (edit: added ETag-Header on 304 response):

const got = require('got');
const http = require('http');
const util = require('util');

const cache = new Map();
http.createServer((req, res) => {
  if (req.headers['if-none-match'] === 'asdf') {
    res.writeHead(304, { 'ETag': 'asdf' });
    return res.end();
  }
  res.writeHead(200, { 'ETag': 'asdf' });
  res.end(Buffer.from('content', 'utf-8'));
})
  .listen(1234, async () => {
    try {
      console.log('Starting request 1...');
      const res1 = await got('http://localhost:1234', { cache });
      console.log('Finished request 1');
      console.dir(res1.body);

      console.log('Starting request 2...');
      const res2 = await got('http://localhost:1234', { cache });
      console.log('Finished request 2');
      console.dir(res2.body);

      console.log('Finished');
      process.exit(0);
    } catch (e) {
      console.error(e);
      process.exit(1);
    }
  });

got@10.7.0 seems to work fine:

Starting request 1...
Finished request 1
'content'
Starting request 2...
Finished request 2
'content'
Finished

got@11.0.3 returns an empty response on the second request:

Starting request 1...
Finished request 1
'content'
Starting request 2...
Finished request 2
''
Finished

got@11.1.4 and got@11.2.0 hangs forever:

Starting request 1...
Finished request 1
'content'
Starting request 2...

Any idea of what’s wrong here or should I open a new issue?

michael42 on Jun 5, 2020

so the cacheableResponse event is never fired.

Then the promise resolves with a Response and _onResponse gets called.

With some “High quality debugging” (console.log) I can say that _onResponse is not called the second time.

Starting request 1...
_onResponse
Finished request 1
'content'
Starting request 2...

I’m going to look into why that doesn’t happen.

Giotino on Jun 5, 2020