scrapy: Scrapy randomly hangs for no reason, then continues a couple of minutes later

Description

Scrapy hangs after x minutes and then continues after x minutes. I found that it stopped when the clock hit xx:xx:25, sends log messages at xx:xx:35, continues at xx:xx:10. I removed Concurrent requests = 3 and now it seems to freeze without a pattern. I am running 12 spiders with crawlerprocess and looping 575 pages.

Reproduces how often:

every x minutes

Logs


2022-02-12 20:35:17 [rotating_proxies.middlewares] DEBUG: Retrying <GET some url>
2022-02-12 20:35:45 [scrapy.extensions.logstats] INFO: Crawled 27 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:35:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2021s)
2022-02-12 20:35:45 [scrapy.extensions.logstats] INFO: Crawled 30 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:35:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 838s)
2022-02-12 20:35:45 [scrapy.extensions.logstats] INFO: Crawled 32 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:35:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2835s)
2022-02-12 20:35:45 [scrapy.extensions.logstats] INFO: Crawled 24 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:35:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 773s)
2022-02-12 20:35:45 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:35:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2122s)
2022-02-12 20:35:45 [scrapy.extensions.logstats] INFO: Crawled 26 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:35:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1676s)
2022-02-12 20:35:45 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:35:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2031s)
2022-02-12 20:35:45 [scrapy.extensions.logstats] INFO: Crawled 25 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:35:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1508s)
2022-02-12 20:35:45 [scrapy.extensions.logstats] INFO: Crawled 27 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:35:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1222s)
2022-02-12 20:35:45 [scrapy.extensions.logstats] INFO: Crawled 22 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:35:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1983s)
2022-02-12 20:35:45 [scrapy.extensions.logstats] INFO: Crawled 23 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:35:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1851s)
2022-02-12 20:35:45 [scrapy.extensions.logstats] INFO: Crawled 28 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:35:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1282s)
2022-02-12 20:36:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2021s)
2022-02-12 20:36:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 838s)
2022-02-12 20:36:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2835s)
2022-02-12 20:36:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 773s)
2022-02-12 20:36:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2122s)
2022-02-12 20:36:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1676s)
2022-02-12 20:36:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2031s)
2022-02-12 20:36:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1508s)
2022-02-12 20:36:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1222s)
2022-02-12 20:36:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1983s)
2022-02-12 20:36:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1851s)
2022-02-12 20:36:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1282s)
2022-02-12 20:36:45 [scrapy.extensions.logstats] INFO: Crawled 27 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:36:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2021s)
2022-02-12 20:36:45 [scrapy.extensions.logstats] INFO: Crawled 30 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:36:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 838s)
2022-02-12 20:36:45 [scrapy.extensions.logstats] INFO: Crawled 32 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:36:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2835s)
2022-02-12 20:36:45 [scrapy.extensions.logstats] INFO: Crawled 24 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:36:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 773s)
2022-02-12 20:36:45 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:36:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2122s)
2022-02-12 20:36:45 [scrapy.extensions.logstats] INFO: Crawled 26 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:36:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1676s)
2022-02-12 20:36:45 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:36:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2031s)
2022-02-12 20:36:45 [scrapy.extensions.logstats] INFO: Crawled 25 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:36:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1508s)
2022-02-12 20:36:45 [scrapy.extensions.logstats] INFO: Crawled 27 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:36:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1222s)
2022-02-12 20:36:45 [scrapy.extensions.logstats] INFO: Crawled 22 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:36:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1983s)
2022-02-12 20:36:45 [scrapy.extensions.logstats] INFO: Crawled 23 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:36:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1851s)
2022-02-12 20:36:45 [scrapy.extensions.logstats] INFO: Crawled 28 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:36:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1282s)
2022-02-12 20:37:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2021s)
2022-02-12 20:37:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 838s)
2022-02-12 20:37:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2835s)
2022-02-12 20:37:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 773s)
2022-02-12 20:37:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2122s)
2022-02-12 20:37:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1676s)
2022-02-12 20:37:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2031s)
2022-02-12 20:37:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1508s)
2022-02-12 20:37:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1222s)
2022-02-12 20:37:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1983s)
2022-02-12 20:37:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1851s)
2022-02-12 20:37:15 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1282s)
2022-02-12 20:37:45 [scrapy.extensions.logstats] INFO: Crawled 27 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:37:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2021s)
2022-02-12 20:37:45 [scrapy.extensions.logstats] INFO: Crawled 30 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:37:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 838s)
2022-02-12 20:37:45 [scrapy.extensions.logstats] INFO: Crawled 32 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:37:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2835s)
2022-02-12 20:37:45 [scrapy.extensions.logstats] INFO: Crawled 24 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:37:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 773s)
2022-02-12 20:37:45 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:37:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2122s)
2022-02-12 20:37:45 [scrapy.extensions.logstats] INFO: Crawled 26 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:37:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1676s)
2022-02-12 20:37:45 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:37:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 2031s)
2022-02-12 20:37:45 [scrapy.extensions.logstats] INFO: Crawled 25 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:37:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1508s)
2022-02-12 20:37:45 [scrapy.extensions.logstats] INFO: Crawled 27 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:37:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1222s)
2022-02-12 20:37:45 [scrapy.extensions.logstats] INFO: Crawled 22 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:37:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1983s)
2022-02-12 20:37:45 [scrapy.extensions.logstats] INFO: Crawled 23 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:37:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1851s)
2022-02-12 20:37:45 [scrapy.extensions.logstats] INFO: Crawled 28 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-02-12 20:37:45 [rotating_proxies.middlewares] INFO: Proxies(good: 0, dead: 2, unchecked: 498, reanimated: 0, mean backoff time: 1282s)
2022-02-12 20:37:49 [rotating_proxies.middlewares] DEBUG: Retrying <GET some url

Minimal, Reproducible Example

import scrapy
from scrapy.crawler import CrawlerProcess
from fake_useragent import UserAgent
from urllib.parse import urlencode
import re
import os
import time
import subprocess
import threading
import datetime
import pickle
import sys
import random
from scrapy.utils.project import get_project_settings
import logging
import codecs

words = []
logger = logging.getLogger('rotating_proxies.expire')
logger.setLevel(logging.DEBUG)
logger = logging.getLogger('rotating_proxies.middleware')
logger.setLevel(logging.DEBUG)

products_list1 = []
products_list2 = []
products_list3 = []
products_list4 = []
products_list5 = []
products_list6 = []
products_list7 = []
products_list8 = []
products_list9 = []
products_list10 = []
products_list11 = []
products_list12 = []

# ---------- ADD ALL WORDS TO LIST ---------- #
with open("SecondaryWords.txt", "r", encoding="utf-8") as wordsfile:
    database = wordsfile.readlines()
    for line in database:
        line = line.strip()
        words.append(line)


def split_list(alist, wanted_parts=1):
    length = len(alist)
    return [alist[i * length // wanted_parts: (i + 1) * length // wanted_parts]
            for i in range(wanted_parts)]

class reset_bad_proxies():
    def __init__(self, min_period_seconds):
        self.min_period = datetime.timedelta(seconds=min_period_seconds * 60)
        self.last_calltime = datetime.datetime.now() - self.min_period

    def perform_reset(self):
        now = datetime.datetime.now()
        if now - self.last_calltime < self.min_period:
            return
        else:
            self.last_calltime = now
            bad_proxy_amount = 0
            for bad_proxy in bad_proxies:
                bad_proxies.remove(bad_proxy)
                good_proxies.append(bad_proxy)
                bad_proxy_amount += 1
            # print(str(bad_proxy_amount) + " Proxy/Proxies Moved To Good Proxies")
            logging.warning("[Custom Rotating Proxy] " + str(bad_proxy_amount) + " Proxy/Proxies Moved To Good Proxies")

templist = split_list(words, wanted_parts=12)
testbot1Words = templist[0]
testbot2Words = templist[1]
testbot3Words = templist[2]
testbot4Words = templist[3]
testbot5Words = templist[4]
testbot6Words = templist[5]
testbot7Words = templist[6]
testbot8Words = templist[7]
testbot9Words = templist[8]
testbot10Words = templist[9]
testbot11Words = templist[10]
testbot12Words = templist[11]

bad_proxy_append = reset_bad_proxies(10)

good_proxies = []
bad_proxies = []

# with codecs.open("MyProxies.txt", 'r', encoding='utf8') as f:
#     good_proxies.append([line.strip() for line in f if line.strip()])

with open("WebsharePaidProxiesFixed.txt", "r", encoding='utf-8') as proxy_file:
    for line in proxy_file:
        if line.strip():
            good_proxies.append(line.strip())

# def reset_bad_proxies():
#     for bad_proxy in bad_proxies:
#         bad_proxies.remove(bad_proxy)
#         good_proxies.append(bad_proxy)

def add_bad_proxy(bad_proxy):
    good_proxies.remove(bad_proxy)
    bad_proxies.append(bad_proxy)
    logging.warning("[Custom Rotating Proxy] " + str(bad_proxy) + " Moved To Dead Proxies")
    # print(str(bad_proxy) + " Moved To Dead Proxies")

class testbot12(scrapy.Spider):
    name = "test"

    def start_requests(self):
        while True:
            random.shuffle(testbot12Words)
            for url in testbot12Words:
                current_proxy = random.choice(good_proxies)
                request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
                request.cb_kwargs['word'] = url
                request.cb_kwargs['proxy'] = str(current_proxy)
                request.meta['proxy'] = str(current_proxy)
                yield request
            bad_proxy_append.perform_reset()

    def parse(self, response, word, proxy):
        if response.status == 200:
            pass
        elif response.status == 404:
            pass
        elif response.status == 503:
            pass
        else:
            add_bad_proxy(str(proxy))
        products_list12.clear()
        for item in response.css('div.col-sm-4 col-lg-4 col-md-4'):
            product_name = item.css('a.title').css('::text').extract_first()
            product_price = item.css('div > div.caption > h4.pull-right.price').css('::text').extract_first()

            allproducts = 'title: ' + product_name + ' ' + "testbot" + ' ' + \
                          'price: ' + product_price
            products_list12.append(allproducts)


class testbot11(scrapy.Spider):
    name = "test"

    def start_requests(self):
        while True:
            random.shuffle(testbot11Words)
            for url in testbot11Words:
                current_proxy = random.choice(good_proxies)
                request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
                request.cb_kwargs['word'] = url
                request.cb_kwargs['proxy'] = str(current_proxy)
                request.meta['proxy'] = str(current_proxy)
                yield request
            bad_proxy_append.perform_reset()

    def parse(self, response, word, proxy):
        if response.status == 200:
            pass
        elif response.status == 404:
            pass
        elif response.status == 503:
            pass
        else:
            add_bad_proxy(str(proxy))
        products_list11.clear()
        for item in response.css('div.col-sm-4 col-lg-4 col-md-4'):
            product_name = item.css('a.title').css('::text').extract_first()
            product_price = item.css('div > div.caption > h4.pull-right.price').css('::text').extract_first()

            allproducts = 'title: ' + product_name + ' ' + "testbot" + ' ' + \
                          'price: ' + product_price
            products_list11.append(allproducts)


class testbot10(scrapy.Spider):
    name = "test"

    def start_requests(self):
        while True:
            random.shuffle(testbot10Words)
            for url in testbot10Words:
                current_proxy = random.choice(good_proxies)
                request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
                request.cb_kwargs['word'] = url
                request.cb_kwargs['proxy'] = str(current_proxy)
                request.meta['proxy'] = str(current_proxy)
                yield request
            bad_proxy_append.perform_reset()

    def parse(self, response, word, proxy):
        if response.status == 200:
            pass
        elif response.status == 404:
            pass
        elif response.status == 503:
            pass
        else:
            add_bad_proxy(str(proxy))
        products_list10.clear()
        for item in response.css('div.col-sm-4 col-lg-4 col-md-4'):
            product_name = item.css('a.title').css('::text').extract_first()
            product_price = item.css('div > div.caption > h4.pull-right.price').css('::text').extract_first()

            allproducts = 'title: ' + product_name + ' ' + "testbot" + ' ' + \
                          'price: ' + product_price
            products_list10.append(allproducts)


class testbot9(scrapy.Spider):
    name = "test"

    def start_requests(self):
        while True:
            random.shuffle(testbot9Words)
            for url in testbot9Words:
                current_proxy = random.choice(good_proxies)
                request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
                request.cb_kwargs['word'] = url
                request.cb_kwargs['proxy'] = str(current_proxy)
                request.meta['proxy'] = str(current_proxy)
                yield request
            bad_proxy_append.perform_reset()

    def parse(self, response, word, proxy):
        if response.status == 200:
            pass
        elif response.status == 404:
            pass
        elif response.status == 503:
            pass
        else:
            add_bad_proxy(str(proxy))
        products_list9.clear()
        for item in response.css('div.col-sm-4 col-lg-4 col-md-4'):
            product_name = item.css('a.title').css('::text').extract_first()
            product_price = item.css('div > div.caption > h4.pull-right.price').css('::text').extract_first()

            allproducts = 'title: ' + product_name + ' ' + "testbot" + ' ' + \
                          'price: ' + product_price
            products_list9.append(allproducts)


class testbot8(scrapy.Spider):
    name = "test"

    def start_requests(self):
        while True:
            random.shuffle(testbot8Words)
            for url in testbot8Words:
                current_proxy = random.choice(good_proxies)
                request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
                request.cb_kwargs['word'] = url
                request.cb_kwargs['proxy'] = str(current_proxy)
                request.meta['proxy'] = str(current_proxy)
                yield request
            bad_proxy_append.perform_reset()

    def parse(self, response, word, proxy):
        if response.status == 200:
            pass
        elif response.status == 404:
            pass
        elif response.status == 503:
            pass
        else:
            add_bad_proxy(str(proxy))
        products_list8.clear()
        for item in response.css('div.col-sm-4 col-lg-4 col-md-4'):
            product_name = item.css('a.title').css('::text').extract_first()
            product_price = item.css('div > div.caption > h4.pull-right.price').css('::text').extract_first()

            allproducts = 'title: ' + product_name + ' ' + "testbot" + ' ' + \
                          'price: ' + product_price
            products_list8.append(allproducts)


class testbot7(scrapy.Spider):
    name = "test"

    def start_requests(self):
        while True:
            random.shuffle(testbot7Words)
            for url in testbot7Words:
                current_proxy = random.choice(good_proxies)
                request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
                request.cb_kwargs['word'] = url
                request.cb_kwargs['proxy'] = str(current_proxy)
                request.meta['proxy'] = str(current_proxy)
                yield request
            bad_proxy_append.perform_reset()

    def parse(self, response, word, proxy):
        if response.status == 200:
            pass
        elif response.status == 404:
            pass
        elif response.status == 503:
            pass
        else:
            add_bad_proxy(str(proxy))
        products_list7.clear()
        for item in response.css('div.col-sm-4 col-lg-4 col-md-4'):
            product_name = item.css('a.title').css('::text').extract_first()
            product_price = item.css('div > div.caption > h4.pull-right.price').css('::text').extract_first()

            allproducts = 'title: ' + product_name + ' ' + "testbot" + ' ' + \
                          'price: ' + product_price
            products_list7.append(allproducts)


class testbot6(scrapy.Spider):
    name = "test"

    def start_requests(self):
        while True:
            random.shuffle(testbot6Words)
            for url in testbot6Words:
                current_proxy = random.choice(good_proxies)
                request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
                request.cb_kwargs['word'] = url
                request.cb_kwargs['proxy'] = str(current_proxy)
                request.meta['proxy'] = str(current_proxy)
                yield request
            bad_proxy_append.perform_reset()

    def parse(self, response, word, proxy):
        if response.status == 200:
            pass
        elif response.status == 404:
            pass
        elif response.status == 503:
            pass
        else:
            add_bad_proxy(str(proxy))
        products_list6.clear()
        for item in response.css('div.col-sm-4 col-lg-4 col-md-4'):
            product_name = item.css('a.title').css('::text').extract_first()
            product_price = item.css('div > div.caption > h4.pull-right.price').css('::text').extract_first()

            allproducts = 'title: ' + product_name + ' ' + "testbot" + ' ' + \
                          'price: ' + product_price
            products_list6.append(allproducts)


class testbot5(scrapy.Spider):
    name = "test"

    def start_requests(self):
        while True:
            random.shuffle(testbot5Words)
            for url in testbot5Words:
                current_proxy = random.choice(good_proxies)
                request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
                request.cb_kwargs['word'] = url
                request.cb_kwargs['proxy'] = str(current_proxy)
                request.meta['proxy'] = str(current_proxy)
                yield request
            bad_proxy_append.perform_reset()

    def parse(self, response, word, proxy):
        if response.status == 200:
            pass
        elif response.status == 404:
            pass
        elif response.status == 503:
            pass
        else:
            add_bad_proxy(str(proxy))
        products_list5.clear()
        for item in response.css('div.col-sm-4 col-lg-4 col-md-4'):
            product_name = item.css('a.title').css('::text').extract_first()
            product_price = item.css('div > div.caption > h4.pull-right.price').css('::text').extract_first()

            allproducts = 'title: ' + product_name + ' ' + "testbot" + ' ' + \
                          'price: ' + product_price
            products_list5.append(allproducts)


class testbot4(scrapy.Spider):
    name = "test"

    def start_requests(self):
        while True:
            random.shuffle(testbot4Words)
            for url in testbot4Words:
                current_proxy = random.choice(good_proxies)
                request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
                request.cb_kwargs['word'] = url
                request.cb_kwargs['proxy'] = str(current_proxy)
                request.meta['proxy'] = str(current_proxy)
                yield request
            bad_proxy_append.perform_reset()

    def parse(self, response, word, proxy):
        if response.status == 200:
            pass
        elif response.status == 404:
            pass
        elif response.status == 503:
            pass
        else:
            add_bad_proxy(str(proxy))
        products_list4.clear()
        for item in response.css('div.col-sm-4 col-lg-4 col-md-4'):
            product_name = item.css('a.title').css('::text').extract_first()
            product_price = item.css('div > div.caption > h4.pull-right.price').css('::text').extract_first()

            allproducts = 'title: ' + product_name + ' ' + "testbot" + ' ' + \
                          'price: ' + product_price
            products_list4.append(allproducts)


class testbot3(scrapy.Spider):
    name = "test"

    def start_requests(self):
        while True:
            random.shuffle(testbot3Words)
            for url in testbot3Words:
                current_proxy = random.choice(good_proxies)
                request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
                request.cb_kwargs['word'] = url
                request.cb_kwargs['proxy'] = str(current_proxy)
                request.meta['proxy'] = str(current_proxy)
                yield request
            bad_proxy_append.perform_reset()

    def parse(self, response, word, proxy):
        if response.status == 200:
            pass
        elif response.status == 404:
            pass
        elif response.status == 503:
            pass
        else:
            add_bad_proxy(str(proxy))
        products_list3.clear()
        for item in response.css('div.col-sm-4 col-lg-4 col-md-4'):
            product_name = item.css('a.title').css('::text').extract_first()
            product_price = item.css('div > div.caption > h4.pull-right.price').css('::text').extract_first()

            allproducts = 'title: ' + product_name + ' ' + "testbot" + ' ' + \
                          'price: ' + product_price
            products_list3.append(allproducts)


class testbot2(scrapy.Spider):
    name = "test"

    def start_requests(self):
        while True:
            random.shuffle(testbot2Words)
            for url in testbot2Words:
                current_proxy = random.choice(good_proxies)
                request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
                request.cb_kwargs['word'] = url
                request.cb_kwargs['proxy'] = str(current_proxy)
                request.meta['proxy'] = str(current_proxy)
                yield request
            bad_proxy_append.perform_reset()

    def parse(self, response, word, proxy):
        if response.status == 200:
            pass
        elif response.status == 404:
            pass
        elif response.status == 503:
            pass
        else:
            add_bad_proxy(str(proxy))
        products_list2.clear()
        for item in response.css('div.col-sm-4 col-lg-4 col-md-4'):
            product_name = item.css('a.title').css('::text').extract_first()
            product_price = item.css('div > div.caption > h4.pull-right.price').css('::text').extract_first()

            allproducts = 'title: ' + product_name + ' ' + "testbot" + ' ' + \
                          'price: ' + product_price
            products_list2.append(allproducts)


class testbot1(scrapy.Spider):
    name = "test"

    def start_requests(self):
        while True:
            random.shuffle(testbot1Words)
            for url in testbot1Words:
                current_proxy = random.choice(good_proxies)
                request = scrapy.Request(url=url, callback=self.parse, dont_filter=True)
                request.cb_kwargs['word'] = url
                request.cb_kwargs['proxy'] = str(current_proxy)
                request.meta['proxy'] = str(current_proxy)
                yield request
            bad_proxy_append.perform_reset()

    def parse(self, response, word, proxy):
        if response.status == 200:
            pass
        elif response.status == 404:
            pass
        elif response.status == 503:
            pass
        else:
            add_bad_proxy(str(proxy))
        products_list1.clear()
        for item in response.css('div.col-sm-4 col-lg-4 col-md-4'):
            product_name = item.css('a.title').css('::text').extract_first()
            product_price = item.css('div > div.caption > h4.pull-right.price').css('::text').extract_first()

            allproducts = 'title: ' + product_name + ' ' + "testbot" + ' ' + \
                          'price: ' + product_price
            products_list1.append(allproducts)


if __name__ == '__main__':
    ua = UserAgent()

    process = CrawlerProcess({
        'DOWNLOAD_DELAY': 1.5,
        # 'ROTATING_PROXY_LIST_PATH': 'WebsharePaidProxiesFixed.txt',
        'ROBOTSTXT_OBEY': False,
        # 'DOWNLOADER_MIDDLEWARES': {
        #     'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
        #     'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
        # },
        # 'DOWNLOADER_MIDDLEWARES': {
        #     'middlewaresproxycustom.TestMiddlewareCustom': 610,
        # },
        # 'ROTATING_PROXY_BAN_POLICY': 'policy.BanPolicy',
        'DEFAULT_REQUEST_HEADERS': {
            'Accept': '*/*',
            'Accept-Encoding': 'gzip, deflate, br',
            'Connection': 'keep-alive',
            'Content-Type': 'text/plain;charset=UTF-8',
        },
        # 'CONCURRENT_REQUESTS': 3,
        # 'CONCURRENT_REQUESTS_PER_DOMAIN': 3,
        # 'FAKEUSERAGENT_PROVIDERS': [
        #     'scrapy_fake_useragent.providers.FakeUserAgentProvider',  # this is the first provider we'll try
        #     'scrapy_fake_useragent.providers.FakerProvider',
        #     # if FakeUserAgentProvider fails, we'll use faker to generate a user-agent string for us
        #     'scrapy_fake_useragent.providers.FixedUserAgentProvider',  # fall back to USER_AGENT value
        # ],
        'USER_AGENT': ua.chrome,
        'LOG_LEVEL': 'DEBUG',
    })

    process.crawl(testbot1)
    process.crawl(testbot2)
    process.crawl(testbot3)
    process.crawl(testbot4)
    process.crawl(testbot5)
    process.crawl(testbot6)
    process.crawl(testbot7)
    process.crawl(testbot8)
    process.crawl(testbot9)
    process.crawl(testbot10)
    process.crawl(testbot11)
    process.crawl(testbot12)
    process.start()  # the script will block here until the crawling is finished

I’m unable to provide an example without using proxies since that seems to be the problem. It turned out it was the scrapy-rotating-proxies middleware that was the problem so I decided to create my own simple test proxy and the same thing happens.

Additional Information

It’s working totally fine when running with, for example, 6 spiders using crawlerprocess, but it hangs when using 12 spiders. Nothing in the code is different other than the amount of spiders. The spiders code are the same but with different variable names which doesn’t make any difference.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 15 (8 by maintainers)

Most upvoted comments

Which means you need to debug this issue yourself.

Please reopen when you have something actionable.