puppeteer: Overcome Cloudflare's blocking: Please turn JavaScript on and reload the page.

Steps to reproduce

Tell us about your environment:

Puppeteer version: 8.0.0
Platform / OS version: Ubuntu 20.04 LTS
URLs (if applicable):
Node.js version: v10.21.0

What steps will reproduce the problem?

    const response = await page.goto(`https://finviz.com/news.ashx`, {
      waitUntil: 'networkidle0'
    });
    console.log(await page.content());

What is the expected result?

Get normal page

What happens instead?

Got,

Please turn JavaScript on and reload the page
Please enable Cookies and reload the page.
This process is automatic. Your browser will redirect to your requested content shortly.
Please allow up to 5 seconds…

            DDoS protection by <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a>

The url has been working fine for ages, until it broke just a day or two before.

Any way to by-pass it? Thx

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (1 by maintainers)

Most upvoted comments

https://www.npmjs.com/package/puppeteer-extra-plugin-stealth might help… 🤞

+15

mrceperka on Mar 26, 2021

Thanks a lot @mrceperka for sharing that well-kept secret, really appreciate it!!! Horay!

+13

suntong on Apr 2, 2021

The only change is from headless: true to headless: false and script would work magically. How strange!

Seems to me that Cloudflare is able to detect when I’m using headless: true and send the alternative blocking html, with Please turn JavaScript on and reload the page.

Even if I trigger page.reload myself won’t help.

Any idea anything else I can try, to make headless: true working?

suntong on Mar 20, 2021

https://www.npmjs.com/package/puppeteer-extra-plugin-stealth might help… 🤞

Worked for me to fix the issue Screen Shot 2022-03-04 at 1 23 43 PM

10111282 on Mar 4, 2022

thanks @suntong and @mrceperka 😄

basetta on Sep 7, 2021

read my last comment again, @mtatarau90. If you don’t get it, read it again, and again, until you get it – what my last sentence and those up-votes means.

That’s all I want to say on this.

suntong on Aug 7, 2021

Nope, only works when headless: false, 😦

suntong on Mar 20, 2021