Scrapegraph-ai: asyncio.run() cannot be called from a running event loop
Hi there,
trying to get SmartScraperGraph running on Fast-API.
@app.post("/crawl")
async def crawl(request: Request):
data = await request.json()
url = data.get('url')
try:
smart_scraper_graph = SmartScraperGraph(
prompt="List me all the articles",
# also accepts a string with the already downloaded HTML code
source=url,
config=graph_config
)
result = smart_scraper_graph.run()
print(result)
# Access the URL field
return result
except Exception as e:
print(f"Error in crawl: {e}")
return None
Config
`graph_config = {
"llm": {
"model": "ollama/llama3",
"temperature": 0,
"format": "json", # Ollama needs the format to be specified explicitly
"base_url": "http://localhost:11434", # set ollama URL arbitrarily
},
"embeddings": {
"model": "ollama/nomic-embed-text",
"base_url": "http://localhost:11434", # set ollama URL arbitrarilyURL
}
}`
Error:
Error in crawl: asyncio.run() cannot be called from a running event loop
/Users/konrad/Documents/Projects/product-spider/apps/service/main.py:171: RuntimeWarning: coroutine 'AsyncChromiumLoader.ascrape_playwright' was never awaited
Any idea? Thanks
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 6
- Comments: 18 (1 by maintainers)
Run this before the above code.
This answer solved my problem.
Encountering this issue too, while trying to run the graph from an async function (in my case a NATS event handler), I found the following workaround.
Basically it executes the asyncio event loop on another thread, but waits for the executing in the current event loop.
Not sure if there are any downsides using this approach, as I am fairly new to working with Python event loops. Looking forward to built-in support
Hey there, yes
Playwrightusesasynciounder the hood so probably you are trying to run an asyncio routine inside another one (your async crawl method). Right now the .rnu method doeesn’t include a way to handle asynchronous calls but since it is a requested feature we will add it 😃Also we will include other web driver like the one provided by Selenium
please add the all the code
Hey - I ran into smth similar while trying to wrap the smart scraper graph with some fastapi endpoints - what worked for me was to wrap the whole thing with run_in_threadpool from starlette.concurrency - running version 1.2.3