puppeteer-extra: [Bug] Sites with hard challenges of CloudFlare do not work with `browser.newPage()` and call `browser.pages()` Disrupted them tabs!
The site https://www.000webhost.com/cpanel-login has a hard challenge for Cloudflare and does not open normal. So I used the puppeteer-extra-plugin-stealth plugin.
This site will not be opened with the browser.newPage(). (The tick we hit again the same challenge page)
But in the first default tab, which is always open, this site opens! (We tick and the site opens.)
This is strange so far, but it gets strange.
So I tried to use the same tab without newPage().
I tried to get the pages first:
var pages = await browser.pages();
Then open the site in the first tab with pages[0].goto.
But this time the site didn’t open! (I mean fails to dissolve challenge Cloudflare)
It looks like it doesn’t open when I call newPage().
And also when the pages() method is called, all tabs for this site are disrupted. (Even the first tab that can normally open this site)
I was confused and I think there’s a bug here.
Sample code:
import puppeteerExtra from 'puppeteer-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';
var puppeteer = puppeteerExtra.use(StealthPlugin());
var browser = await puppeteer.launch({ headless: false });
// const page = await browser.newPage();
var pages = await browser.pages();
pages[0].goto('https://www.000webhost.com/cpanel-login');
Versions:
node v19.7.0
puppeteer@21.1.1
puppeteer-extra@3.3.6
puppeteer-extra-plugin-stealth@2.11.2
Chrome Version 116.0.5845.141
Video:
https://github.com/berstend/puppeteer-extra/assets/246721/f527b517-5fb5-4941-8c81-1bd37d9b1046
About this issue
- Original URL
- State: open
- Created 10 months ago
- Reactions: 1
- Comments: 36
@mowatermelon Thanks for your answer.
It wasn’t bad as a temporary trick. But it has problems and of course the bug still exists.
By setting
userDataDir, we can maintain the previous status and after solving the challenge manually, the next time Chrome opens, we can open the site without the challenge page.But when the CloudFlare cookie session expires, everything goes back to the way it was before. If we use, for example,
newPage(), the challenge will not be solved. Or if we reach the first tab with the help ofpages()[0], everything is broken there and the site’s challenge is not solved in any way.Unless we repeat the trick again, i.e. temporarily remove the call to
pages()and run Chrome once to pass the challenge. Then return that function to our code.In general, it did not cure much pain!
I can confirm that it works also on version “puppeteer”: “^9.1.1”. Anything higher than that does not work for me
has anyone found solutions to this? I can’t bypass captcha on cloudflare anymore
Unfortunately, this problem was very serious and acute. And I had to use the service FlareSolverr. This is a proxy to bypass cloudflare and they use selenium. I just send my first page to it and return only the
cf_clearancecookie, set it to my puppeteer and continue…you guys can back to puppeteer version 5.5.0, all the problems will be solved, however, it quite old
I have tried the constant page refresh problem when trying to switch to Cloudflare on the aforementioned sites. I have been repeatedly signing up on a different site for about 10 hours and they use cloudflare premium captcha. It passed all of them without any problem without needing to touch it. I don’t see any problem in the package right now except fingerprint. Could you please try the package? I tried it on many sites like 000webhost openai etc. It was successful on all of them.
@Hillcow @joeledwardson @ergcode @NabiKAZ @wlc108 @mowatermelon
https://www.npmjs.com/package/puppeteer-real-browser
I had this problem too, so I had to find a solution and publish it. I found a way to make the browser look real. I connected to the browser with Puppeteer. I exported the browser and page created with Puppeteer. You can use all the functions you use with Puppeteer here. Proxy supported.