knightcrawler: Scarping 1337x error

I got from logs:

[Thu Jan 25 2024 20:55:44 GMT+0000] starting 1337x scrape...
Scrapping 1337x Movies category page 1
Failed 1337x scrapping for [1] Movies due:  Error: Failed browse request
    at browse (/home/node/app/scrapers/1337x/1337x_api.js:66:27)
    at /home/node/app/scrapers/1337x/1337x_api.js:79:23
Scrapping 1337x TV category page 1
Failed 1337x scrapping for [1] TV due:  Error: Failed browse request
    at browse (/home/node/app/scrapers/1337x/1337x_api.js:66:27)
    at /home/node/app/scrapers/1337x/1337x_api.js:79:23
Scrapping 1337x Anime category page 1
Failed 1337x scrapping for [1] Anime due:  Error: Failed browse request
    at browse (/home/node/app/scrapers/1337x/1337x_api.js:66:27)
    at /home/node/app/scrapers/1337x/1337x_api.js:79:23
Scrapping 1337x Documentaries category page 1
Failed 1337x scrapping for [1] Documentaries due:  Error: Failed browse request
    at browse (/home/node/app/scrapers/1337x/1337x_api.js:66:27)
    at /home/node/app/scrapers/1337x/1337x_api.js:79:23
[Thu Jan 25 2024 20:55:44 GMT+0000] finished 1337x scrape

someone knows what should I do to solve it?

About this issue

Original URL
State: closed
Created 5 months ago
Comments: 32 (22 by maintainers)

Most upvoted comments

I forgot about Flaresolver. I had it setup briefly for Jackett as well but went a different direction. I was playing around with some headful browsers in containers but…

curl -L -X POST 'http://192.168.1.2:8191/v1' \
-H 'Content-Type: application/json' \
--data-raw '{
  "cmd": "request.get",
  "url":"https://1337x.to/torrent/2099267/Ubuntu-MATE-16-04-2-MATE-armhf-img-xz-Uzerus/",
  "maxTimeout": 60000
}'
{"status": "ok", "message": "Challenge solved!", "solution": {"url": "https://1337x.to/torrent/2099267/Ubuntu-MATE-16-04-2-MATE-armhf-img-xz-Uzerus/", "status": 200, "cookies": [{"domain": ".1337x.to", "expiry": 1737790044, "httpOnly": true, "name": "cf_clearance", "path": "/", "sameSite": "None", "secure": true,...

Looks like we might have a path forward! This was ran on a residential IP, not a datacenter IP for what it’s worth.

davidjameshowell on Jan 26, 2024

I don’t think it is necessary to use multiple instances nor to use sessions. What my solution is doing is: Make a request using Flaresolverr Once the request passes the cloudflare protection, we get, additionally to the normal response, the cookie from Cloudflare, which we can use to make requests that don’t have to go through the cloudflare protection, thus not needing Flaresolverr.

One thing I think needs improvement in my solution is error handling. Let’s say we get an error because the cookies are not valid anymore, we should reset them and try to go the route with Flaresolverr again.

I will maybe add this later or tomorrow.

KillTrot on Jan 31, 2024

So I have been running into issues with Flaresolverr. It does not like to be bombarded with requests and will silently stop solving if you send a bunch of requests to FS. Individual requests seem to work OK (without sessions), so I’ll try to investigate slowing requests down or waiting for success/failures first.

davidjameshowell on Jan 28, 2024

[Fri Jan 26 2024 17:52:31 GMT+0000] starting 1337x scrape...
Scrapping 1337x Movies category page 1
(node:11870) [MONGODB DRIVER] DeprecationWarning: Db.collection option [strict] is deprecated and will be removed in a later version.
(Use `node --trace-deprecation ...` to show where the warning was created)
Created 1337x entry for [XXXX] The Continental From the XXX of John XXX 2023 S01 XXX WEBRip SDR 10Bit 1440p DDP5.1 Atmos HEVC-3Li
Created 1337x entry for [XXXX] Obitelj.Thornberry.Film.(2002).XXXX.x265.4Mbps.2CH.320.crtani.film.hrvatski.sink

Using Flaresolverr has worked for me. Will continue scraping 1337x and see where that lands.

davidjameshowell on Jan 26, 2024

I created a pull request #19

KillTrot on Jan 31, 2024