FlareSolverr: Captchas not working

With todays cloudflare patch our 2 solutions don’t seem to work anymore.

Working implementation of this 3rd patch for undetect_chromedriver (single python file):
https://github.com/digitalnomad91/udetect_coudflare_working_patch/

Here is the latest version of flaresolverr with this fix implemented: 
https://github.com/digitalnomad91/FlareSolverr-udetect-patch/

Note: I haven’t tested running it through a new docker image / container, but it should work just fine. I’ll include a systemd service implementation for those that want to run it standalone.

Still waiting on this (seemingly) more robust method to be implemented & released:
https://github.com/ultrafunkamsterdam/undetected-chromedriver/discussions/1420

I also put together a small browser extension that can run in Chrome or Firefox that worked well enough to get me through the periods where CF was winning. Will probably be useful for if/when there are no known solutions like right now:
https://github.com/digitalnomad91/cloudflare-bypass-extension

Done: #842

I agree with that, it would benefit everyone if you could make a pull request from this issue

Well I fixed my 403 issue, so I can make a PR if everyone wants, but I don’t think it will be accepted as the maintainers probably don’t want to just comment out some of the things I did.

I can fork and make a PR on the official branch anyways though, sec… I’m pretty sure I just made some changes to src/utils.py.

I can also confirm I am experiencing the same issue, even after updating docker image to the latest 3.2.2 hotfix.

log from flaresolver: 2023-07-22 12:17:41 INFO Incoming request => POST /v1 body: {'maxTimeout': 55000, 'cmd': 'request.get', 'url': 'https://ipt.lol/t?73=&26=&55=&78=&23=&24=&25=&66=&65=&79=&22=&5=&99=&4=&o=time'} 2023-07-22 12:17:45 INFO Challenge detected. Title found: Just a moment... 2023-07-22 12:18:41 ERROR Error: Error solving the challenge. Timeout after 55.0 seconds. 2023-07-22 12:18:41 INFO Response in 60.119 s 2023-07-22 12:18:41 INFO 172.17.0.7 POST http://172.17.0.10:8191/v1 500 Internal Server Error

log from jackett: 07-22 13:29:18 Info Torznab search in TorrentGalaxy => Found 50 releases (from cache) [0ms] 07-22 13:29:36 Info CardigannIndexer(torrentgalaxy): Relogin required 07-22 13:29:36 Info CardigannIndexer(torrentgalaxy): Relogin required 07-22 13:29:41 Error CardigannIndexer (torrentgalaxy): Found captcha during automatic login, aborting 07-22 13:29:41 Error Error on 20230722132941 for TorrentGalaxy.txtSystem.Exception: Relogin failed at Jackett.Common.Indexers.CardigannIndexer.PerformQuery(TorznabQuery query) in ./Jackett.Common/Indexers/CardigannIndexer.cs:line 1719

Replace /base_library.zip/urllib/parse.pyc with working v3.2.1 version.

I assumed the Linux version was working because the Docker version was working, but I’m realising now that was stupid, they’re not the same. ngosang is working on v3.3.0, so I won’t waste time putting out another hotfix.

Just the dev tools was not enough to get around this issue when I tested. Opened this PR which seems to resolve it for me, running on windows.

It’s Patch Tuesday so yeah, has to be an update. When you disable headless mode you can see it clicks the verify you are a human checkbox but then CF just shows another one, and another one, etc. Manually clicking the checkbox or interacting with the page doesn’t help. But I did notice something interesting. While it’s stuck solving the turnstiles, if you open a new tab manually and then load the same URL it will solve the challenge in the new tab, which then also solves it for the first tab. I tried recreating this behavior programmatically but it only seems to work if the tab was manually created from the UI. Seems like they know how to detect non user-controlled tabs.

I just tested and it’s stll working for me on my hetzner machines and @ home. If you are triggering an actual captcha then I don’t believe flaresolverr can defeat that atm. Perhaps something like this could work & be integrated in somehow:

https://addons.mozilla.org/en-US/firefox/addon/buster-captcha-solver/

I’m not sure if it’s able to solve CF captchas, but if so then it could be useful in the webextension hacky workaround I shared as well. Fortunately for my use case I am not running into actual captchas, just the CF redirect check if you’re human thing.

@tupac55 @damien2405 @MrAlucardDante try this if you want to run through docker:

git clone https://github.com/digitalnomad91/FlareSolverr && cd FlareSolverr
git checkout cloudflare_fix_3
docker build --file Dockerfile -t digitalnomad91/flaresolverr:cloudflare_fix_3 .
docker rm -f flaresolverr
docker run -d   --name=flaresolverr  -p 8191:8191  -e LOG_LEVEL=info  --restart unless-stopped  digitalnomad91/flaresolverr:cloudflare_fix_3
docker logs flaresolverr -f

@YasserAntonio When I tried it with your captcha url I got an error like this:

2023-08-02 01:53:10 INFO     Incoming request => POST /v1 body: {'cmd': 'request.get', 'url': 'https://www.opensubtitles.org/en/captcha/redirect-%7Cen%7Cdownload%7Csub%7C641843', 'maxTimeout': 60000}
2023-08-02 01:53:11 ERROR    Error: Error solving the challenge. Message: unknown error: cannot determine loading status\nfrom disconnected: Unable to receive message from renderer\n  (Session info: chrome=115.0.5790.98)\nStacktrace:\n#0 0x55b8a2f624e3 <unknown>\n#1 0x55b8a2c91c76 <unknown>\n#2 0x55b8a2c7b284 <unknown>\n#3 0x55b8a2c7afa0 <unknown>\n#4 0x55b8a2c799bf <unknown>\n#5 0x55b8a2c7a162 <unknown>\n#6 0x55b8a2c8854b <unknown>\n#7 0x55b8a2c89222 <unknown>\n#8 0x55b8a2c9bbcb <unknown>\n#9 0x55b8a2ca03ab <unknown>\n#10 0x55b8a2c7a703 <unknown>\n#11 0x55b8a2c9b810 <unknown>\n#12 0x55b8a2d05523 <unknown>\n#13 0x55b8a2cecde3 <unknown>\n#14 0x55b8a2cc22dd <unknown>\n#15 0x55b8a2cc334e <unknown>\n#16 0x55b8a2f223e4 <unknown>\n#17 0x55b8a2f263d7 <unknown>\n#18 0x55b8a2f30b20 <unknown>\n#19 0x55b8a2f27023 <unknown>\n#20 0x55b8a2ef51aa <unknown>\n#21 0x55b8a2f4b6b8 <unknown>\n#22 0x55b8a2f4b847 <unknown>\n#23 0x55b8a2f5b243 <unknown>\n#24 0x7f007f43eea7 start_thread\n

I don’t think flaresolverr is able to handle any captchas at the moment (at least if the warning in the readme is to be believed). Thankfully my use case hasn’t ran into that problem (yet), but for what it’s worth I was able to successfully get a response with the direct download link from the HTML like this:

curl -L -X POST 'http://127.0.0.1:8191/v1' \
-H 'Content-Type: application/json' \
--data-raw '{
 "cmd": "request.get",
 "url":"https://www.opensubtitles.org/en/subtitleserve/sub/9642256",
 "maxTimeout": 60000
}'

Maybe take a look at the link in my comment right before this one as the captcha it shows me looks like it’s one that can be defeated by that webextension ^. I’m not sure if or how difficult it would be to combine that with flaresolverr / undetected-chromedriver, but I bet it could be combined relatively easily with the hacky webextension solution I shared up above. It’s still pretty hacky and I’ve only ran it in a desktop GUI, but it worked for getting me through the last couple weeks where there was no working solution. If/when CF is winning again then I will likely go back to using it again, so if/when that happens I will try to clean it up a bit and make some improvements (and perhaps adding that captcha solver plugin into it). I initially just ran it at home and had a couple others do the same, but you could easily use something like cockpit to setup a VM with a desktop gui that you can access right through your web browser. Then you can just leave that running on your server and point whatever process you need to consume the data from towards that webserver.js /read route in the repo. Good luck!

I agree with that, it would benefit everyone if you could make a pull request from this issue

Well I fixed my 403 issue, so I can make a PR if everyone wants, but I don’t think it will be accepted as the maintainers probably don’t want to just comment out some of the things I did.

I can fork and make a PR on the official branch anyways though, sec… I’m pretty sure I just made some changes to src/utils.py.

You can always try to make a pull request, it doesn’t cost anything to try, anyway, as you said, it’s always a cat and mouse game to add a workaround to cloudflare’s limitations, so I think whatever the solution is if it’s functional, the repo owner should welcome it ^^

I agree with that, it would benefit everyone if you could make a pull request from this issue

@digitalnomad91 why not make a pull request? From what I understand, you’ve make a fork of this repository, so if its functional the maintener of flaresolverr can just add straight away your changes ?

@antivirtel My mistake I forgot to update the repo url, should be good now!

https://github.com/digitalnomad91/FlareSolverr-udetect-patch/

Windows hotfix now available - https://github.com/FlareSolverr/FlareSolverr/releases/v3.2.2-hotfix

Q: how to easily get either of these in my install?

The method shown in PR https://github.com/FlareSolverr/FlareSolverr/pull/812/files worked for me.

I added the line at exactly the position shown in the PR.

ultrafunkamsterdam/undetected-chromedriver#1388

There’s a workaround in here (open dev tools on the tab)

https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/1388

Same issue here. CF just shows an infinite amount of turnstiles in the challenges. Looks like it might be fingerprinting since it doesn’t happen outside of Selenium.

Yep, they definitely updated something. Hopefully we can get a resolution soon. Also, I just realized the warning in the README.md was from a while ago, so not related to this issue at all.