crawl4ai: Error using crawler: maximum recursion depth exceeded

I’m getting the maximum recursion depth exceeded error in the crawling application I have. See portion of the trace below (I’m only including a portion because the whole thing is insanely long, and the lines repeat over and over again, so not too useful).

Some important context:

  • The application is running in a Docker container
  • Within the app, there are 2 processes: one is triggered to occur every 1 hour; the second - every 1 minute.

What I tried:

  • I set thread_safe to True for both cases

Otherwise, I do not known how I can debug this.

I suspect that the issue could be caused by 2 crawl processes overlapping, which is supported by bigger picture I see in the logs, and specifically, that the 1-minute job starts and finishes successfully several times until the longer-running 1-hour job starts. At this point, the 1-minute job would hang, and after some time fail with lost connectivity message. Subsequent 1-minute jobs return the maximum recursion depth exceeded error. Would you say this suspicion is valid?

Example trace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/wapaganda/webparse/source_update_service.py", line 38, in run_update
    new_yield = await self.scanner.fetch_items()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/wapaganda/webparse/sources/infox_ru.py", line 57, in fetch_items
    res = await self.fetch_page_items(url)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/wapaganda/webparse/sources/scanner.py", line 59, in fetch_page_items
    async with AsyncWebCrawler(thread_safe=True) as crawler:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_webcrawler.py", line 218, in __aenter__
    return await self.start()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_webcrawler.py", line 203, in start
    await self.awarmup()
  File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_webcrawler.py", line 232, in awarmup
    self.logger.info(f"Crawl4AI {crawl4ai_version}", tag="INIT")
  File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_logger.py", line 158, in info
    self._log(LogLevel.INFO, message, tag, **kwargs)
  File "/usr/local/lib/python3.12/site-packages/crawl4ai/async_logger.py", line 147, in _log
    print(log_line)
  File "/usr/local/lib/python3.12/site-packages/colorama/ansitowin32.py", line 47, in write
    self.__convertor.write(text)
  File "/usr/local/lib/python3.12/site-packages/colorama/ansitowin32.py", line 177, in write
    self.write_and_convert(text)
  File "/usr/local/lib/python3.12/site-packages/colorama/ansitowin32.py", line 202, in write_and_convert
    self.write_plain_text(text, cursor, start)
  File "/usr/local/lib/python3.12/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text
    self.wrapped.write(text[start:end])
  File "/usr/local/lib/python3.12/site-packages/colorama/ansitowin32.py", line 47, in write
    self.__convertor.write(text)
  File "/usr/local/lib/python3.12/site-packages/colorama/ansitowin32.py", line 177, in write

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 30 (10 by maintainers)

Most upvoted comments

Still same issue:

Image

@betterthanever2 In just a few days, I will release a new version, and we have something very interesting. You’re going to love it. It is designed to handle, and encapsulates all the difficulties of crawling theoretically infinite URLs simultaneously, depending on the available memory resources. I will record a good video and post it on my X account.

@betterthanever2 Please try version ~0.4.248~ 0.4.24y. I fixed the memory leak issue. Even with the current Docker, you shouldn’t encounter any problems. I suggest you try the code below first to get yourself familiar. This is part of new documentation that I will upload soon:

import os
import sys
import psutil
import asyncio

__location__ = os.path.dirname(os.path.abspath(__file__))
__output__ = os.path.join(__location__, "output")

# Append parent directory to system path
parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.append(parent_dir)

from typing import List
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def crawl_parallel(urls: List[str], max_concurrent: int = 3):
    print("\n=== Parallel Crawling with Browser Reuse + Memory Check ===")

    # We'll keep track of peak memory usage across all tasks
    peak_memory = 0
    process = psutil.Process(os.getpid())

    def log_memory(prefix: str = ""):
        nonlocal peak_memory
        current_mem = process.memory_info().rss  # in bytes
        if current_mem > peak_memory:
            peak_memory = current_mem
        print(f"{prefix} Current Memory: {current_mem // (1024 * 1024)} MB, Peak: {peak_memory // (1024 * 1024)} MB")

    # Minimal browser config
    browser_config = BrowserConfig(
        headless=True,
        verbose=False,   
        extra_args=["--disable-gpu", "--disable-dev-shm-usage", "--no-sandbox"],
    )
    crawl_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

    # Create the crawler instance
    crawler = AsyncWebCrawler(config=browser_config)
    await crawler.start()

    try:
        # We'll chunk the URLs in batches of 'max_concurrent'
        success_count = 0
        fail_count = 0
        for i in range(0, len(urls), max_concurrent):
            batch = urls[i : i + max_concurrent]
            tasks = []

            for j, url in enumerate(batch):
                # Unique session_id per concurrent sub-task
                session_id = f"parallel_session_{i + j}"
                task = crawler.arun(url=url, config=crawl_config, session_id=session_id)
                tasks.append(task)

            # Check memory usage prior to launching tasks
            log_memory(prefix=f"Before batch {i//max_concurrent + 1}: ")

            # Gather results
            results = await asyncio.gather(*tasks, return_exceptions=True)

            # Check memory usage after tasks complete
            log_memory(prefix=f"After batch {i//max_concurrent + 1}: ")

            # Evaluate results
            for url, result in zip(batch, results):
                if isinstance(result, Exception):
                    print(f"Error crawling {url}: {result}")
                    fail_count += 1
                elif result.success:
                    success_count += 1
                else:
                    fail_count += 1

        print(f"\nSummary:")
        print(f"  - Successfully crawled: {success_count}")
        print(f"  - Failed: {fail_count}")

    finally:
        print("\nClosing crawler...")
        await crawler.close()
        # Final memory log
        log_memory(prefix="Final: ")
        print(f"\nPeak memory usage (MB): {peak_memory // (1024 * 1024)}")

async def main():
    urls = [
        f"https://example.com/page{i}" for i in range(1, 21)
    ]
    await crawl_parallel(urls, max_concurrent=4)

if __name__ == "__main__":
    asyncio.run(main())

Take a look at the output:

=== Parallel Crawling with Browser Reuse + Memory Check ===
Before batch 1:  Current Memory: 131 MB, Peak: 131 MB
After batch 1:  Current Memory: 133 MB, Peak: 133 MB
Before batch 2:  Current Memory: 133 MB, Peak: 133 MB
After batch 2:  Current Memory: 134 MB, Peak: 134 MB
Before batch 3:  Current Memory: 134 MB, Peak: 134 MB
After batch 3:  Current Memory: 134 MB, Peak: 134 MB
Before batch 4:  Current Memory: 134 MB, Peak: 134 MB
After batch 4:  Current Memory: 135 MB, Peak: 135 MB
Before batch 5:  Current Memory: 135 MB, Peak: 135 MB
After batch 5:  Current Memory: 135 MB, Peak: 135 MB

Summary:
  - Successfully crawled: 20
  - Failed: 0

Closing crawler...
Final:  Current Memory: 135 MB, Peak: 135 MB

Peak memory usage (MB): 135

As you can see the memory peak has been 135MB.

@betterthanever2 Ok, I can see what we can do. By tomorrow or day after tomorrow, I will write a simple script that improves the crawling process. Additionally, I am updating all the documents, as some have become outdated. I made changes in version 0.4 that caused contradictions in the documents, and I am addressing all of them. Hopefully, we will resolve this within a few weeks plus releasing the new executor and docker. In the meantime, tomorrow or the day after, I will provide a short solution to help you handle it better.

@psychicDivine @Liky98 @betterthanever2 Sorry, my mistake. You can already upgrade to version 0.4.247. It was a typo mistake when I mentioned 0.4.248, so if you install down the latest version pip install -U crawl4ai I hope everything works fine and there shouldn’t be any memory leakage anymore. Please try it and let me know.

@unclecode Thank you for the reply! I mistyped the version when testing the fix, everything’s OK now. Looking forward to the new release!

@unclecode I think that specific issue occurred because I tried to crawl concurrently too many pages at once - with properly organized code, I can’t replicate it. Thank you for the assist!

@betterthanever2 Glad to hear that, and regarding the second issue, are you able to replicate it, so I can focus on that? Please try it and then make a new issue.

Thank you for addressing the issue and offering a solution. I sincerely hope that the memory leak issue will be resolved moving forward.

Thank you! Will try this asap.

@unclecode thank you for your response The thing is, as far as I can tell there is very little difference between my code and what I saw in the documentation. Here’s an example method

    async def fetch_page_items(self, url: str) -> list[dict] | None:
        async with AsyncWebCrawler(verbose=True) as crawler:
            result = await crawler.arun(url=url,
                                        extraction_strategy=self.extraction_strategy,
                                        bypass_cache=True)
        assert result.success
        logfire.debug(f"Content {result.model_dump()} items")
        items = json.loads(result.extracted_content)
        return items

Certain options can be different (cache use, verbosity etc), but all in all nothing outlandish. I am sure you are correct about the numerous instances of browser being launched, but I haven’t seen this in the docs.

Remember, there are different ways to manage browser instances. You can create a new browser instance or lock one browser instance and create new pages within that instance.

Can you point me in the direction of some documentation that describes how to do this?