SeleniumBase: Suddenly unable to bypass CloudFlare challenge (Ubuntu Server)

Hello, overnight my instances of seleniumbase became unable to bypass the CloudFlare challenge ( which uses CloudFlare turnstile ).

I was using an older version of SB so I updated to latest ( 4.27.4 ), and it is still not passing the challenge.

cloudflare_chal

I am using your demo code for clicking on the CloudFlare turnstile captcha:

from seleniumbase import SB

def open_the_turnstile_page(sb):
    url = "https://wildbet.gg/"
    sb.driver.uc_open_with_reconnect(url, reconnect_time=5)

def click_turnstile_and_verify(sb):
    sb.switch_to_frame("iframe")
    sb.driver.uc_click("span")
    sb.assert_element("img#captcha-success", timeout=3)

with SB(uc=True, test=True) as sb:
    open_the_turnstile_page(sb)
    try:
        click_turnstile_and_verify(sb)
    except Exception:
        open_the_turnstile_page(sb)
        click_turnstile_and_verify(sb)
    sb.set_messenger_theme(location="top_left")
    sb.post_message("SeleniumBase wasn't detected", duration=3)

if I instead use: sb.driver.uc_open_with_reconnect(url, reconnect_time=9999)

and click manually, it works. This means they are detecting something ?

I also tried adding reconnect_time=5 on uc_click and it did not help.

I’m a big fan of your project and I’ve been using it for some time 😃

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 73 (27 by maintainers)

Most upvoted comments

More details:

Now, if your User-Agent looks untrustworthy, CloudFlare makes you click the CAPTCHA (which has been improved). If they detect either Selenium in the browser or JavaScript involvement in clicking the CAPTCHA, they don’t let the click through. That’s why pyautogui is now required for clicking the CAPTCHA if the User-Agent isn’t trustworthy enough. The default user-agent set on macOS and Windows by SeleniumBase is generally good enough. On Linux, the default User-Agent might not be good enough: You may need to specify a better one to avoid needing to click the CAPTCHA in that scenario. (Or just use the pyautogui workaround for clicking it… Scroll up to see some examples that use it.)

I’m working on an update that can optionally utilize the pyautogui workaround if needed. That will likely need an update to examples because the existing uc_click might not be good enough if the User-Agent isn’t trustworthy enough.

This probably means a new UC Mode Video Tutorial (Part 3) is likely to happen soon to explain the changes.

This was resolved in 4.28.0 - https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.28.0

Read https://github.com/seleniumbase/SeleniumBase/issues/2865 for all the details. You may need to use the new UC Mode methods in 4.28.0, such as driver.uc_gui_handle_cf(), in order to successfully click through CF CAPTCHA checkboxes on Linux.

OK, I’m trying to figure out why uc_click(selector) behaves differently in different environments (Linux vs macOS/Windows). I’ll hopefully have an update ready soon / early this week.

If anyone figures out the reason for the uc_click(selector) differences before I do, let me know. The good news is that sb.driver.uc_open_with_reconnect(url, reconnect_time) is working correctly, and that’s the main component of UC Mode.

Hopefully there’s a pyautogui alternative to clicking in a stealthy way. Currently, that is an option. Eg. If you use driver.uc_open_with_disconnect(url), then you can use pyautogui to click in a stealthy way, followed by driver.connect() so that you can perform more Selenium actions after that.

The pyautogui.press(" ") solution works consistently as well under Linux. You can make it work in headless mode, and in a multi-process environment, by attaching it to a virtual display. This code worked for me under Ubuntu/ Debian (note that headed=True; but Selenium will run anyway headless)…

from seleniumbase import SB
import pyautogui
from pyvirtualdisplay.display import Display
disp = Display(visible=True, size=(1366, 768), backend="xvfb", use_xauth=True)
disp.start()

import Xlib.display
pyautogui._pyautogui_x11._display = Xlib.display.Display(os.environ['DISPLAY'])

with SB(uc=True, headed=True) as sb:
     ...

OK, here’s the full scope:

UC Mode: –> macOS: ✅ (All good) –> Windows: ✅ (All good) –> Linux (Ubuntu / server): ⁉️ (conditional) ------> uc_open_with_reconnect(): ✅ ----------> Manual actions (while disconnected): ✅ ----------> pyautogui actions (while disconnected): ✅ ----------> uc_click(selector): ❌

I’ll be working on either fixing uc_click(selector) on Linux, or an alternative way to successfully clicking a CAPTCHA checkbox on Linux.

@JimKarvo pyautogui won’t work for headless browsers, or multithreaded, but if you wanted to see an example script that uses pyautogui, here’s one:

import pyautogui
from seleniumbase import SB

with SB(uc=True, test=True, incognito=True) as sb:
    sb.driver.uc_open_with_reconnect("https://seleniumbase.io/apps/turnstile")
    sb.send_keys("html", "\t")
    sb.driver.disconnect()
    pyautogui.press(" ")
    sb.sleep(1)
    sb.driver.connect()
    sb.assert_element("img#captcha-success", timeout=3)
    sb.set_messenger_theme(location="top_left")
    sb.post_message("SeleniumBase wasn't detected", duration=3)

Obviously, due to those pyautogui limitations, I want a better workaround for Ubuntu Servers. At least macOS and Windows are still good for now. (And yes, I’m working on Ubuntu solutions.)

Hello @mdmintz , I also had the issue yesterday at all my VMs, Not only Linux (Ubuntu 22.04, Debian 11), but also in a Windows Server 2022, It works perfectly only in my personal laptop having Windows 11 Home, Basically: At VMs:

  • it never passes without displaying the CF checkbox challenge
  • uc_click seems to be detected, and it never recovers
  • on the same SB browser, when clicking manually instead of letting SB clicks, it works pretty quick after the manual click
  • tried with or without proxy, the same results
  • versions: google-chrome version 125.0.6422.142 under Windows Server, google-chrome version 124.0.6367.118 under Ubuntu 22.04, google-chrome version 122.0.6261.111 under Debian 11. I also tried going in SeleniumBase version as old as v4.23.0, v4.20.8, but it didn’t help (same blockage behavior).

At my personal laptop (Windows 11):

  • most of the time, it passes without having to click anything
  • when it has to click, it passes as well
  • tried with or without proxy (same proxy that’s used for the VMs) -> always working fine
  • versions: google-chrome version 125.0.6422.142

It looks like Cloudflare is able to tell if I’m using a VM somehow, but yet, it works when clicking manually… I can say it can be quickly reproduced on any Google Compute Engine VM.

I hope my description is helpful, And thanks for the great project ^^

@ismayilibrahimov Azure has a known IP-range (just like AWS or GCP). That’s why residential proxies have become so popular lately for web-scraping.

I test https://github.com/sarperavci/CloudflareBypassForScraping the script uses the DrissionPage When you open the browser normally, it automatically succeeds without clicking to confirm that you are a human. When verify you are human appears, he clicks on the box and success is given. Can you merge or modify uc_click to work on the same principle on the Seleniumbase

got it working guys, implemented the crazy solution, Launched 2 threads, 1 thread will handle seleniumbase and another will handle pyautogui and we are good to go

This will not work at headless systems although

I found something interesting. If I do the following on a Mac (with a Linux UA), and click manually before the reconnect_time expires, then the CAPTCHA is bypassed successfully. But if I wait for the uc_click(selector) to happen, then it fails:

from seleniumbase import SB

agent=("Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.2 (KHTML, like Gecko) Chrome/94.0.4096.99 Safari/534.2 EdgA/94.01135.87")

with SB(uc=True, test=True, agent=agent) as sb:
    url = "https://visa.vfsglobal.com/fra/en/hrv/login"
    sb.driver.uc_open_with_reconnect(url, 15)
    sb.switch_to_frame("iframe")
    sb.driver.uc_click("span")
    breakpoint()

So as long as Selenium is not detected during the clicking part, it doesn’t seem like they cared that there was a Linux UA for a macOS system. (But note that the website will still block you if they detect a bad UA string, such as one that contains HeadlessChrome in it.)

@OpsecGuy You seem to be the only one having Windows issues at this time. You also introduced a lot of variables into the equation in your example. Eg. 1. using Driver() instead of SB(). 2. Using Dark Mode. 3. Using any headless mode. 4. Changing proxy settings. 5. Adding an extension. You should stick with the specific examples I tried in this thread (or another UC Mode example from the SeleniumBase/examples folder) for debugging purposes.