scrapy: raise CloseSpider("Error Message") is not saving error message
When I run the code raise CloseSpider("Error Message") in the Spider parse function, the reason gets saved in the finish_reason stats string.
When it is run in the start_requests function, the spider properly closes but the finish_reason does not have the message it in. The finish_reason value is “finished” instead of “Error Message” as set in the CloseSpider exception.
Is this intentional or a bug? thanks!
About this issue
- Original URL
- State: open
- Created 8 years ago
- Reactions: 2
- Comments: 16 (8 by maintainers)
Commits related to this issue
- Resolved #3463 — committed to shinhonggwan/scrapy by shinhonggwan 2 years ago
If someone needs a workaround to raise a
CloseSpiderwith a specific message in thestart_requestsmethod before the fix is released, we did something like this:Of course, you can change the try/except by a if/else or whatever you want and the “url” by any other URL. The “magic” here occurs when calling the callback, as the reason of the
CloseSpiderwill be respected.@Gallaecio I beg to differ, but I don’t think this is a bug, since the docs do say the exception “can be raised from a spider callback”. Any change to allow raising the exception from other components should be considered an enhancement IMHO.
Hi, could me and my team tackle this issue ?
Any progress on the related pull request?
I’m trying to detect the reason on spider closed signal listener and getting “finished” no mater how the close is called:
Hoping this would be fixed with #4058