puppeteer: Page timeout due to script loading error/timeout

In my case the goto timeout happens because of a forever-loading blocking resource (js or css). That’ll never trigger the page’s load or domcontentloaded. A bug in Puppeteer IMO, but whatever.

My fix (FINALLY!) is to do what Lighthouse does in its Driver: a Promise.race() for a custom ‘timeout’-ish. The shorter version I used:

const LOAD_FAIL = Math.random();
const sleep = options => new Promise(resolve => {
	options.timer = setTimeout(resolve, options.ms, options.result === undefined ? true : options.result);
});

const sleepOptions = {ms: TIMEOUT - 1000, result: LOAD_FAIL};
const response = await Promise.race([
	sleep(sleepOptions),
	page.goto(url, {timeout: TIMEOUT + 1000}),
]);
clearTimeout(sleepOptions.timer);
const success = response !== LOAD_FAIL;

The sleep() and page.goto() race. If it’s the timeout, the result is clearly a fail (LOAD_FAIL). Otherwise it’s whatever page.goto() returns. Like this, it doesn’t throw a TimeoutException, and the page is still queryable.

That was my issue. I don’t know all your semi-related issues.

Is the normal behavior of puppeteer not supposed to be the same as a “normal” browser ? In this case I think this issue should be reopen.

@sorxrob I found the solution. Just add to launch args - --enable-blink-features=HTMLImports

browser = await puppeteer.launch({
            //headless: false,
            'args': [
                '--enable-blink-features=HTMLImports'
            ]
        });

https://magpcss.org/ceforum/viewtopic.php?f=6&t=17412

Actually, chrome has a 1 minute timeout for pending requests, so the following works fine for me:

const puppeteer = require('puppeteer');

(async() => {
    const browser = await puppeteer.launch({
      headless: false,
    });
    const [page] = await browser.pages();
    await page.goto('https://www.noordwijk.nl/', {
      timeout: 90000,
    });
    await browser.close();
})();

@kingiw I can’t reproduce your issue. On my machine it just work ok. Have you tried run this with proxy VPN? Maybe you are banned. The log should be longer.

200 https://www.google.com/images/searchbox/desktop_searchbox_sprites302_hr.webp
200 https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png
200 https://www.google.com/xjs/_/js/k=xjs.s.pl.nOBV0DuCEwY.O/ck=xjs.s.ofAvoXrQJ80.L.W.O/m=Fkg7bd,HcFEGb,IvlUe,MC8mtf,OF7gzc,RMhBfe,T4BAC,TJw5qb,TbaHGc,Y33vzc,cdos,cr,hsm,iDPoPb,jsa,mvYTse,tg8oTe,uz938c,vWNDde,ws9Tlc,yQ43ff,d,csi/am=AAAAAEsAdt0BAP43ARAA2DEAACBAmGBjgTBIKAhArJoABAI/d=1/dg=2/ct=zgms/rs=ACT90oEalR9CJZXl8zSGPMqdX4dMWrsj9A
200 https://ssl.gstatic.com/gb/images/i1_1967ca6a.png
200 https://www.gstatic.com/images/branding/googlemic/2x/googlemic_color_24dp.png
200 https://www.gstatic.com/inputtools/images/tia.png
204 https://www.google.com/gen_204?s=webhp&t=aft&atyp=csi&ei=5RhiXsmSDcakad3FoDA&rt=wsrt.462,aft.206,prt.185&bl=ED6e&ima=1&imad=0&imn=2
200 https://www.google.com/images/nav_logo299.webp
204 https://www.google.com/gen_204?atyp=csi&ei=5RhiXsmSDcakad3FoDA&s=jsa&jsi=s,t.0,et.focus,n.iDPoPb,cn.1&zx=1583487207037
200 https://www.google.com/xjs/_/js/k=xjs.s.pl.nOBV0DuCEwY.O/ck=xjs.s.ofAvoXrQJ80.L.W.O/am=AAAAAEsAdt0BAP43ARAA2DEAACBAmGBjgTBIKAhArJoABAI/d=1/exm=Fkg7bd,HcFEGb,IvlUe,MC8mtf,OF7gzc,RMhBfe,T4BAC,TJw5qb,TbaHGc,Y33vzc,cdos,cr,csi,d,hsm,iDPoPb,jsa,mvYTse,tg8oTe,uz938c,vWNDde,ws9Tlc,yQ43ff/ed=1/dg=2/ct=zgms/rs=ACT90oEalR9CJZXl8zSGPMqdX4dMWrsj9A/m=RqxLvf,aa,abd,async,dvl,fEVMic,foot,lu,m,mUpTid,mu,sb_wiz,sf,spch,xz7cCd?xjs=s1
200 https://www.gstatic.com/og/_/js/k=og.og2.en_US.ojePM5yvxrU.O/rt=j/m=def,aswid/exm=in,fot/d=1/ed=1/rs=AA2YrTtRGH9zpODrmC5Dv5OxrtsJsy1ZsQ
200 https://www.google.com/xjs/_/js/k=xjs.s.pl.nOBV0DuCEwY.O/ck=xjs.s.ofAvoXrQJ80.L.W.O/am=AAAAAEsAdt0BAP43ARAA2DEAACBAmGBjgTBIKAhArJoABAI/d=1/exm=Fkg7bd,HcFEGb,IvlUe,MC8mtf,OF7gzc,RMhBfe,RqxLvf,T4BAC,TJw5qb,TbaHGc,Y33vzc,aa,abd,async,cdos,cr,csi,d,dvl,fEVMic,foot,hsm,iDPoPb,jsa,lu,m,mUpTid,mu,mvYTse,sb_wiz,sf,spch,tg8oTe,uz938c,vWNDde,ws9Tlc,xz7cCd,yQ43ff/ed=1/dg=2/ct=zgms/rs=ACT90oEalR9CJZXl8zSGPMqdX4dMWrsj9A/m=OG6ZHd,T7XTS,URQPYc,eN4qad,o02Jie,pB6Zqd,zbML3c?xjs=s2
204 https://www.google.com/gen_204?atyp=i&zx=1583487207327&ogsr=1&ei=5RhiXqf_DaqclwSrs5qADA&ct=6&cad=i&id=19016262&loc=webhp&prid=1&ogd=com&ogprm=up&ic=1
200 https://apis.google.com/_/scs/abc-static/_/js/k=gapi.gapi.en.OfYsKuVZ3qI.O/m=gapi_iframes,googleapis_client,plusone/rt=j/sv=1/d=1/ed=1/rs=AHpOoo8UDq_6isr1vipw5cUlPTPPdx3_0A/cb=gapi.loaded_0
200 https://ogs.google.com/widget/app/so?hl=pl&origin=https%3A%2F%2Fwww.google.com&pid=1&spid=1&gm&usegapi=1
204 https://www.google.com/gen_204?atyp=csi&ei=5RhiXsmSDcakad3FoDA&s=webhp&t=all&bl=ED6e&imn=2&adh=&conn=onchange&ima=1&imad=0&ime=1&imex=1&imeh=1&imea=0&imeb=0&wh=600&scp=0&rt=xjsls.173,iml.206,prt.185,aft.206,dcl.187,xjses.327,xjsee.374,xjs.374,ol.999,wsrt.462,cst.116,dnst.1,rqst.284,rspt.158,sslt.67,rqstt.336,unt.217,cstt.219,dit.648&zx=1583487207656
204 https://adservice.google.com/adsid/google/ui

You can try also this https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth to imitate headfull Chrome. The implementation is just:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

If I try on https://try-puppeteer.appspot.com/

const browser = await puppeteer.launch(); // same with {headless: false}, if that works there

const page = await browser.newPage();
await page.goto('https://www.noordwijk.nl/', {timeout: 90000});

console.log(await page.content());
await page.screenshot({path: 'screenshot.png'});

await browser.close();

it errors after a while:

Error running your code. TimeoutError: Navigation Timeout Exceeded: 90000ms exceeded

because the entire page times out, instead of the 1 broken resource, so this entire page is unloadable. Not so in the real browser.

https://try-puppeteer.appspot.com/ doesn’t work either, for either URL. Same timeout (but after the default 30s).