puppeteer: `Navigation Timeout Exceeded` when using `networkidle0` and no insight into what timed out

I suppose an array of the remaining open network connections would at least enable the developer to eliminate certain possibilities but it could also lead to a lot of wild goose chases when none of those URLs are the cause.

You can track this with the request, requestfailed and requestfinished events.

class InflightRequests {
  constructor(page) {
    this._page = page;
    this._requests = new Set();
    this._onStarted = this._onStarted.bind(this);
    this._onFinished = this._onFinished.bind(this);
    this._page.on('request', this._onStarted);
    this._page.on('requestfinished', this._onFinished);
    this._page.on('requestfailed', this._onFinished);
  }

  _onStarted(request) { this._requests.add(request); }
  _onFinished(request) { this._requests.delete(request); }
 
  inflightRequests() { return Array.from(this._requests); }  

  dispose() {
    this._page.removeListener('request', this._onStarted);
    this._page.removeListener('requestfinished', this._onFinished);
    this._page.removeListener('requestfailed', this._onFinished);
  }
}

and use it later like this:

const tracker = new InflightRequests(page);
await page.goto(url).catch(e => {
  console.log('Navigation failed: ' + e.message);
  const inflight = tracker.inflightRequests();
  console.log(inflight.map(request => '  ' + request.url()).join('\n'));
});
tracker.dispose();

Hope this helps.

I have exactly the same problem, the thing is I don’t have this problem if I run puppeteer on Mac, but when I switch to docker (linux) then I have this issue. Both Mac and docker are using the same chrome version HeadlessChrome/66.0.3347.0

@jacobweber something like this might help, unless your sockjs-node calls are required for your page to load correctly:

      await page.setRequestInterception(true)

      page.on('request', req => {
        // disable webpack HMR, which breaks the 'networkidle0' setting
        if (req.url().endsWith('/__webpack_hmr')) {
          req.abort()
        } else {
          req.continue()
        }
      })

I’m getting this Navigation Timeout Exceeded error seemingly randomly. Even using networkidle2. Using 1.0 in docker.

I request we re-open this.

You can track this with the request, requestfailed and requestfinished events.

said @aslushnikov

In https://github.com/peterbe/minimalcss/pull/199 we tried implementing a tracker so that we can extend the Navigation Timeout Exceeded: 30000ms exceeded error message with a list of URLs that got started but never finished. Even with that, and with upgrading to puppeteer 1.4.0, it still fails with the tracker not having any unfinished URLs.

I also tried logging every single possible event (that has to do with requests) on the Page object. And that too didn’t help. Still no insight into what request is exceeding the timeout.

My hunch is that it’s related to service workers. The service-worker set up the site I’m testing against is “stock create-react-app” style. Nothing fancy. I can try the same URLs by changing it to http://songsearch.local which is my local Nginx. When I do that, it always works. And since it’s not HTTPS the service-worker will be automatically disabled.

So, are there tools and techniques to track (and even disable) service workers?

Thanks @aslushnikov, in my case Webpack’s Hot Module Reload remained connected: http://localhost:8009/__webpack_hmr

My limited experience of Navigation Timeout Exceeded errors is that you can get them for a huge number of reasons. Some sites seem to have multiple open connections that even networkidle2 won’t allow for. If you go for the load event and test the same sites, this will tell you whether any given site has this characteristic. Also, timeout exceeded error will occur seemingly randomly where you are operating at the limit of your system resource limitations, whether that be in terms of your memory usage or network connections.