scrapy: raise CloseSpider("Error Message") is not saving error message

When I run the code raise CloseSpider("Error Message") in the Spider parse function, the reason gets saved in the finish_reason stats string.

When it is run in the start_requests function, the spider properly closes but the finish_reason does not have the message it in. The finish_reason value is “finished” instead of “Error Message” as set in the CloseSpider exception.

Is this intentional or a bug? thanks!

About this issue

  • Original URL
  • State: open
  • Created 8 years ago
  • Reactions: 2
  • Comments: 16 (8 by maintainers)

Commits related to this issue

Most upvoted comments

If someone needs a workaround to raise a CloseSpider with a specific message in the start_requests method before the fix is released, we did something like this:

def start_requests(self):
    try:
        # whatever you need to do 
    except:
        yield Request(url='file:///dev/null', callback=self.force_fail, errback=self.force_fail)
        return


def force_fail(self, response):
    raise CloseSpider(reason='failed')

Of course, you can change the try/except by a if/else or whatever you want and the “url” by any other URL. The “magic” here occurs when calling the callback, as the reason of the CloseSpider will be respected.

@Gallaecio I beg to differ, but I don’t think this is a bug, since the docs do say the exception “can be raised from a spider callback”. Any change to allow raising the exception from other components should be considered an enhancement IMHO.

Hi, could me and my team tackle this issue ?

Any progress on the related pull request?

I’m trying to detect the reason on spider closed signal listener and getting “finished” no mater how the close is called:

def spider_closed(self, spider, reason):
        print(reason)

>>> finished

Hoping this would be fixed with #4058