SeleniumBase: Suddenly unable to bypass CloudFlare challenge (Ubuntu Server)
Hello, overnight my instances of seleniumbase became unable to bypass the CloudFlare challenge ( which uses CloudFlare turnstile ).
I was using an older version of SB so I updated to latest ( 4.27.4 ), and it is still not passing the challenge.
I am using your demo code for clicking on the CloudFlare turnstile captcha:
from seleniumbase import SB
def open_the_turnstile_page(sb):
url = "https://wildbet.gg/"
sb.driver.uc_open_with_reconnect(url, reconnect_time=5)
def click_turnstile_and_verify(sb):
sb.switch_to_frame("iframe")
sb.driver.uc_click("span")
sb.assert_element("img#captcha-success", timeout=3)
with SB(uc=True, test=True) as sb:
open_the_turnstile_page(sb)
try:
click_turnstile_and_verify(sb)
except Exception:
open_the_turnstile_page(sb)
click_turnstile_and_verify(sb)
sb.set_messenger_theme(location="top_left")
sb.post_message("SeleniumBase wasn't detected", duration=3)
if I instead use:
sb.driver.uc_open_with_reconnect(url, reconnect_time=9999)
and click manually, it works. This means they are detecting something ?
I also tried adding reconnect_time=5 on uc_click and it did not help.
I’m a big fan of your project and I’ve been using it for some time 😃
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 73 (27 by maintainers)
More details:
Now, if your User-Agent looks untrustworthy, CloudFlare makes you click the CAPTCHA (which has been improved). If they detect either Selenium in the browser or JavaScript involvement in clicking the CAPTCHA, they don’t let the click through. That’s why
pyautoguiis now required for clicking the CAPTCHA if the User-Agent isn’t trustworthy enough. The default user-agent set on macOS and Windows by SeleniumBase is generally good enough. On Linux, the default User-Agent might not be good enough: You may need to specify a better one to avoid needing to click the CAPTCHA in that scenario. (Or just use thepyautoguiworkaround for clicking it… Scroll up to see some examples that use it.)I’m working on an update that can optionally utilize the
pyautoguiworkaround if needed. That will likely need an update to examples because the existinguc_clickmight not be good enough if the User-Agent isn’t trustworthy enough.This probably means a new UC Mode Video Tutorial (Part 3) is likely to happen soon to explain the changes.
This was resolved in
4.28.0- https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.28.0Read https://github.com/seleniumbase/SeleniumBase/issues/2865 for all the details. You may need to use the new UC Mode methods in
4.28.0, such asdriver.uc_gui_handle_cf(), in order to successfully click through CF CAPTCHA checkboxes on Linux.OK, I’m trying to figure out why
uc_click(selector)behaves differently in different environments (Linux vs macOS/Windows). I’ll hopefully have an update ready soon / early this week.If anyone figures out the reason for the
uc_click(selector)differences before I do, let me know. The good news is thatsb.driver.uc_open_with_reconnect(url, reconnect_time)is working correctly, and that’s the main component of UC Mode.Hopefully there’s a
pyautoguialternative to clicking in a stealthy way. Currently, that is an option. Eg. If you usedriver.uc_open_with_disconnect(url), then you can usepyautoguito click in a stealthy way, followed bydriver.connect()so that you can perform more Selenium actions after that.The pyautogui.press(" ") solution works consistently as well under Linux. You can make it work in headless mode, and in a multi-process environment, by attaching it to a virtual display. This code worked for me under Ubuntu/ Debian (note that headed=True; but Selenium will run anyway headless)…
OK, here’s the full scope:
UC Mode: –> macOS: ✅ (All good) –> Windows: ✅ (All good) –> Linux (Ubuntu / server): ⁉️ (conditional) ------>
uc_open_with_reconnect(): ✅ ----------> Manual actions (while disconnected): ✅ ---------->pyautoguiactions (while disconnected): ✅ ---------->uc_click(selector): ❌I’ll be working on either fixing
uc_click(selector)on Linux, or an alternative way to successfully clicking a CAPTCHA checkbox on Linux.@JimKarvo
pyautoguiwon’t work for headless browsers, or multithreaded, but if you wanted to see an example script that usespyautogui, here’s one:Obviously, due to those
pyautoguilimitations, I want a better workaround for Ubuntu Servers. At least macOS and Windows are still good for now. (And yes, I’m working on Ubuntu solutions.)Hello @mdmintz , I also had the issue yesterday at all my VMs, Not only Linux (Ubuntu 22.04, Debian 11), but also in a Windows Server 2022, It works perfectly only in my personal laptop having Windows 11 Home, Basically: At VMs:
At my personal laptop (Windows 11):
It looks like Cloudflare is able to tell if I’m using a VM somehow, but yet, it works when clicking manually… I can say it can be quickly reproduced on any Google Compute Engine VM.
I hope my description is helpful, And thanks for the great project ^^
@ismayilibrahimov Azure has a known IP-range (just like AWS or GCP). That’s why residential proxies have become so popular lately for web-scraping.
I test https://github.com/sarperavci/CloudflareBypassForScraping the script uses the DrissionPage When you open the browser normally, it automatically succeeds without clicking to confirm that you are a human. When verify you are human appears, he clicks on the box and success is given. Can you merge or modify uc_click to work on the same principle on the Seleniumbase
This will not work at headless systems although
I found something interesting. If I do the following on a Mac (with a Linux UA), and click manually before the
reconnect_timeexpires, then the CAPTCHA is bypassed successfully. But if I wait for theuc_click(selector)to happen, then it fails:So as long as Selenium is not detected during the clicking part, it doesn’t seem like they cared that there was a Linux UA for a macOS system. (But note that the website will still block you if they detect a bad UA string, such as one that contains
HeadlessChromein it.)@OpsecGuy You seem to be the only one having Windows issues at this time. You also introduced a lot of variables into the equation in your example. Eg. 1. using
Driver()instead ofSB(). 2. Using Dark Mode. 3. Using any headless mode. 4. Changing proxy settings. 5. Adding an extension. You should stick with the specific examples I tried in this thread (or another UC Mode example from theSeleniumBase/examplesfolder) for debugging purposes.