crawl4ai: Error using crawler: maximum recursion depth exceeded
I’m getting the maximum recursion depth exceeded error in the crawling application I have. See portion of the trace below (I’m only including a portion because the whole thing is insanely long, and the lines repeat over and over again, so not too useful).
Some important context:
- The application is running in a Docker container
- Within the app, there are 2 processes: one is triggered to occur every 1 hour; the second - every 1 minute.
What I tried:
- I set
thread_safeto True for both cases
Otherwise, I do not known how I can debug this.
I suspect that the issue could be caused by 2 crawl processes overlapping, which is supported by bigger picture I see in the logs, and specifically, that the 1-minute job starts and finishes successfully several times until the longer-running 1-hour job starts. At this point, the 1-minute job would hang, and after some time fail with lost connectivity message. Subsequent 1-minute jobs return the maximum recursion depth exceeded error. Would you say this suspicion is valid?
Example trace:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/wapaganda/webparse/source_update_service.py", line 38, in run_update
new_yield = await self.scanner.fetch_items()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/wapaganda/webparse/sources/infox_ru.py", line 57, in fetch_items
res = await self.fetch_page_items(url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/wapaganda/webparse/sources/scanner.py", line 59, in fetch_page_items
async with AsyncWebCrawler(thread_safe=True) as crawler:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_webcrawler.py", line 218, in __aenter__
return await self.start()
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_webcrawler.py", line 203, in start
await self.awarmup()
File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_webcrawler.py", line 232, in awarmup
self.logger.info(f"Crawl4AI {crawl4ai_version}", tag="INIT")
File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_logger.py", line 158, in info
self._log(LogLevel.INFO, message, tag, **kwargs)
File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_logger.py", line 147, in _log
print(log_line)
File "/usr/local/lib/python3.12/site-packages/colorama/ansitowin32.py", line 47, in write
self.__convertor.write(text)
File "/usr/local/lib/python3.12/site-packages/colorama/ansitowin32.py", line 177, in write
self.write_and_convert(text)
File "/usr/local/lib/python3.12/site-packages/colorama/ansitowin32.py", line 202, in write_and_convert
self.write_plain_text(text, cursor, start)
File "/usr/local/lib/python3.12/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text
self.wrapped.write(text[start:end])
File "/usr/local/lib/python3.12/site-packages/colorama/ansitowin32.py", line 47, in write
self.__convertor.write(text)
File "/usr/local/lib/python3.12/site-packages/colorama/ansitowin32.py", line 177, in write
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 30 (10 by maintainers)
Still same issue:
@betterthanever2 In just a few days, I will release a new version, and we have something very interesting. You’re going to love it. It is designed to handle, and encapsulates all the difficulties of crawling theoretically infinite URLs simultaneously, depending on the available memory resources. I will record a good video and post it on my X account.
@betterthanever2 Please try version ~0.4.248~ 0.4.24y. I fixed the memory leak issue. Even with the current Docker, you shouldn’t encounter any problems. I suggest you try the code below first to get yourself familiar. This is part of new documentation that I will upload soon:
Take a look at the output:
As you can see the memory peak has been 135MB.
@betterthanever2 Ok, I can see what we can do. By tomorrow or day after tomorrow, I will write a simple script that improves the crawling process. Additionally, I am updating all the documents, as some have become outdated. I made changes in version 0.4 that caused contradictions in the documents, and I am addressing all of them. Hopefully, we will resolve this within a few weeks plus releasing the new executor and docker. In the meantime, tomorrow or the day after, I will provide a short solution to help you handle it better.
@psychicDivine @Liky98 @betterthanever2 Sorry, my mistake. You can already upgrade to version 0.4.247. It was a typo mistake when I mentioned 0.4.248, so if you install down the latest version
pip install -U crawl4aiI hope everything works fine and there shouldn’t be any memory leakage anymore. Please try it and let me know.@unclecode Thank you for the reply! I mistyped the version when testing the fix, everything’s OK now. Looking forward to the new release!
@unclecode I think that specific issue occurred because I tried to crawl concurrently too many pages at once - with properly organized code, I can’t replicate it. Thank you for the assist!
@betterthanever2 Glad to hear that, and regarding the second issue, are you able to replicate it, so I can focus on that? Please try it and then make a new issue.
Thank you for addressing the issue and offering a solution. I sincerely hope that the memory leak issue will be resolved moving forward.
Thank you! Will try this asap.
@unclecode thank you for your response The thing is, as far as I can tell there is very little difference between my code and what I saw in the documentation. Here’s an example method
Certain options can be different (cache use, verbosity etc), but all in all nothing outlandish. I am sure you are correct about the numerous instances of browser being launched, but I haven’t seen this in the docs.
Can you point me in the direction of some documentation that describes how to do this?