crawl4ai: NotImplementedError in asyncio.create_subprocess_exec on Windows
Issues with crawl4ai Library
1. NotImplementedError in asyncio.create_subprocess_exec on Windows
Description
The crawl4ai library uses asyncio.create_subprocess_exec to start the Playwright browser, which is not supported on Windows. This results in a NotImplementedError, preventing the AsyncWebCrawler from functioning correctly on Windows platforms.
Steps to Reproduce
- Run the code on a Windows platform.
- Use the
AsyncWebCrawlerto crawl a URL. - Observe the
NotImplementedErrorwhen starting the Playwright browser.
Expected Behavior
The AsyncWebCrawler should start the Playwright browser and crawl the URL without raising a NotImplementedError.
Actual Behavior
The code raises a NotImplementedError when attempting to start the Playwright browser using asyncio.create_subprocess_exec.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 26 (14 by maintainers)
@unclecode Thanks for looking in the issue. I tried your given solution and still the same
NotImplementedErrorerror occurs. I tried with both crawl4ai version 0.4.0 and 0.3.71( as it seemed a bit more stable when testing the script). As @hitesh22rana mentioned the issue actually lies with the uvicorn regarding how it starts sub_processes for windows .But even the fix he provided is useful for other machine but not in my case as uvloop neither comes with uvicorn nor i could install it via pip as it’s not even supported for windows machine again ,lol.
What I found finally is that when using uvicorn to start the server I can avoid using the
--reloadflag which actually fixes the problem but then I’ve to restart the server after every change as without --reload flag it doesn’t looks for changes.For now switched to using nodemon in my development environment. And the solution i found for now only works with crawl4ai==0.3.71 version. I tried it using with 0.4.0 but even with this solution of not using the
--reloadflag theNotImplementedErrorerror still continues.For now I’ll stick with the 0.3.71 as it seems helpful and more stable for my use case. Thanks to both of you for looking in the issue @unclecode @hitesh22rana.
@Aniket1026 Regarding the first part, your script works well on my side and this is the output:
For the FastAPI server, remember that the
urlyou pass to thearun()function should be a string, and you are passingHttpUrl. Following is the full code, after a few modifications that work fine on my machine; please try it and let me know. @hitesh22rana thx for support.I tested like this:
In 0.4.1, I am adding this step to raise and error if
urlis not string, I saw similar issue with another case, so better to make it explicit.hey I solved NotImplementedError in windows
import asyncio from asyncio import ProactorEventLoop from crawl4ai import AsyncWebCrawler, CacheMode import streamlit as st
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
st.title(“crawl4AI”) if “crawl_result” not in st.session_state: st.session_state[“crawl_result”] = “”
async def main(): async with AsyncWebCrawler(verbose=True) as crawler: result = await crawler.arun(url =“https://www.inven.co.kr/board/lostark/6271?my=chuchu”) print(result.markdown) st.session_state[“crawl_result”] = result.markdown st.write(st.session_state[“crawl_result”])
if name == “main”: asyncio.run(main())
change your event_loop_policy ProactorEventLoopPolicy()
This is how the script which I used to test looks like and it works fine with crawl4ai==0.3.71 version. As with latest version 0.4.0 it does not even works . You can test this script using any url of an amazon product. command to run : python filename
Then for my use case I tried using it with fastapi in my post route which looks like below: You can start the uvicorn server and send a post request using raw json with “url” as key and any amazon product url as value; command to start server : uvicorn filename:app --reload
With this code i get the error when i hit the post request via postman which is below
my dependencies looks like this
OS - windows10 machine python - 3.10.4 pip - 22.0.4
If there’s something more i could provide you, please let me know if there’s something I could help you with . I hope this will help you generate the same error
I see, yes most likely you are right, thx for explanation @hitesh22rana
Oh, I meant! @skywolf123 might be running their own
fastapiserver locally with thereloadflag enabled. From there, they could be invoking a function via an endpoint, which in turn callscrawl4ai.This doesn’t seem to have anything to do with
crawl4aidirectly, though, as the--reloadflag is likely part of their local setup rather than the Dockerized FastAPI server.@Aniket1026 You’re very welcome, no worries! I definitely want to ensure you get a good response with 0.4.x. Please share as many details about your platform as you can, so I can try to simulate it and reproduce the issue myself.
@Aniket1026,
This seems to be related to an issue with
uvicorn. I found the following reference that might be helpful: GitHub Issue #964The error occurs because
FastAPIusesuvloop, andasynciodoesn’t automatically recognize this without explicitly setting a policy. There’s a helpful answer that outlines hooks to achieve this: StackOverflow threadPlease try to set the following in your code:
Hope this helps! 🙇🏼
Hi @hitesh22rana Thank you for trying it out and coming up with a suggestion. But even with the suggested changes I’m still encountering the same issue. I believe you have a different OS and this could be the reason that you don’t see the same behaviour.
Hi @Aniket1026, cc @unclecode
I couldn’t reproduce the exact behavior you mentioned, but when following your approach, I encountered the following error:
This issue seems to be resolved by modifying the code as shown below:
You can handle URL validations afterward to ensure correctness.
My Environment:
This might be more of a Windows-specific issue, though. If someone using Windows could confirm, that would be helpful! 🙇🏼
Thanks!
@Aniket1026 Hello, thank you for using Crawl for AI! Could you please share your exact code or snippet with me?
When I tried it on my Windows machine, I didn’t encounter this issue, so I’d like to investigate further. Please also share the specs of your operating system, Python version, and any other details you think might help. Once I have that, I’ll try to replicate the error on my end.