cloudflare-scrape: cfscrape cannot solve captchas

Hello! Is there any chance that somebody can help me? The server is a fresh instance set up specially for the test. Ubuntu 19.04 with nodejs installed. None of the python versions(2.7, 3.7) seems to solve the issue. Thanks for any help! Just in case I’ve attached the output from terminal Selection_002

Version number

Run pip show cfscrape and paste the output below: Name: cfscrape Version: 2.0.7 Summary: A simple Python module to bypass Cloudflare’s anti-bot page. See https://github.com/Anorov/cloudflare-scrape for more information. Home-page: https://github.com/Anorov/cloudflare-scrape Author: Anorov Author-email: anorov.vorona@gmail.com License: UNKNOWN Location: /usr/local/lib/python2.7/dist-packages Requires: requests Required-by:

Code snippet experiencing the issue

import cfscrape

scraper = cfscrape.create_scraper() print scraper.get(“https://www.enotes.com/topics/alpha/”).content

Complete exception and traceback

Traceback (most recent call last): File “test.py”, line 5, in <module> print (scraper.get(“https://www.enotes.com/topics/alpha/”).content) # => “<!DOCTYPE html><html><head>…” File “/usr/local/lib/python2.7/dist-packages/requests/sessions.py”, line 546, in get return self.request(‘GET’, url, **kwargs) File “/usr/local/lib/python2.7/dist-packages/cfscrape/init.py”, line 124, in request self.handle_captcha_challenge(resp, url) File “/usr/local/lib/python2.7/dist-packages/cfscrape/init.py”, line 147, in handle_captcha_challenge raise CloudflareCaptchaError(error, response=resp) cfscrape.CloudflareCaptchaError: Cloudflare captcha challenge presented for www.enotes.com (cfscrape cannot solve captchas)

URL of the Cloudflare-protected page

https://www.enotes.com/topics/alpha/

URL of Pastebin/Gist with HTML source of protected page

https://pastebin.com/CXKapc0B

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15

Most upvoted comments

@pro-src I tried my script again after upgrading my OpenSSL from ‘1.1.1.a’ to ‘1.1.1.c’, and got the same captcha result, but after putting the ‘DEFAULT_CIPHERS’ line you recommended for @bajburtskii, it fixed my captcha issue

# import cfscrape

# One TLS v1.3 cipher and one TLS v1.2 cipher
cfscrape.DEFAULT_CIPHERS = 'TLS_AES_256_GCM_SHA384:ECDHE-ECDSA-AES256-SHA384'

# scraper = cfscrape.create_scraper()
# print(scraper.get('https://www.enotes.com/topics/alpha/').text)

956MB on Jun 25, 2019

Hey @bajburtskii,

I hate to ask the obvious here but based on the title… Do you get a CAPTCHA when visiting the URL in your web browser? cfscrape can’t solve CAPTCHA as noted in the README.

Sorry, if that seems like a dumb question but if so, we can close this issue.

ghost on Jun 25, 2019

The difference is in the signature algorithms extensions:

Signature algorithms hash: 94a2ca79b6ef36b1d692cbbfa5641952

Signature algorithms as reported by ssllabs (Mine)

(u'SHA256/ECDSA',
 u'SHA384/ECDSA',
 u'SHA512/ECDSA',
 u'Ed25519',
 u'Ed448',
 u'Unknown (0x8)/Unknown (0x9)',
 u'Unknown (0x8)/Unknown (0xa)',
 u'Unknown (0x8)/Unknown (0xb)',
 u'RSA_PSS_SHA256',
 u'RSA_PSS_SHA384',
 u'RSA_PSS_SHA512',
 u'SHA256/RSA',
 u'SHA384/RSA',
 u'SHA512/RSA',
 u'SHA224/ECDSA',
 u'SHA224/RSA',
 u'SHA224/DSA',
 u'SHA256/DSA',
 u'SHA384/DSA',
 u'SHA512/DSA')

Signature algorithms hash: 402014b899136c3fed09cd745dc01355

Signature algorithms as reported by ssllabs (Yours)

('SHA256/ECDSA',
 'SHA384/ECDSA',
 'SHA512/ECDSA',
 'Ed25519',
 'Ed448',
 'Unknown (0x8)/Unknown (0x9)',
 'Unknown (0x8)/Unknown (0xa)',
 'Unknown (0x8)/Unknown (0xb)',
 'RSA_PSS_SHA256',
 'RSA_PSS_SHA384',
 'RSA_PSS_SHA512',
 'SHA256/RSA',
 'SHA384/RSA',
 'SHA512/RSA',
 'SHA224/ECDSA',
 'SHA1/ECDSA',
 'SHA224/RSA',
 'SHA1/RSA',
 'SHA224/DSA',
 'SHA1/DSA',
 'SHA256/DSA',
 'SHA384/DSA',
 'SHA512/DSA')

Diff:

('SHA1/ECDSA',
 'SHA1/RSA',
 'SHA1/DSA')

Unfortunately, I don’t know of a way to fix this, only suggestions. I find it odd that none of the workarounds worked… I would try using a proxy to rule out an IP problem. Try upgrading openssl to v1.1.1c

ghost on Jun 25, 2019

Sorry @bajburtskii

I’m unable to reproduce the problem, please see https://github.com/Anorov/cloudflare-scrape/issues/264 for a possible workaround.

You might also try setting only a couple of ciphers to avoid the CAPTCHA:

import cfscrape
# One TLS v1.3 cipher and one TLS v1.2 cipher
cfscrape.DEFAULT_CIPHERS = 'TLS_AES_256_GCM_SHA384:ECDHE-ECDSA-AES256-SHA384'

scraper = cfscrape.create_scraper()
print(scraper.get('https://www.enotes.com/topics/alpha/').text)

Otherwise for further assistance, feel free to generate a report with this script.

This must be ran from within the cloned cloudflare-scrape repo if you haven’t pip installed cfscrape

git clone https://gist.github.com/pro-src/17654ec3f949b0b17bd1a4aa1b4136b9 temp
cp temp/report-enotes.py report.py

python2 report.py
python3 report.py

ghost on Jun 25, 2019