Scrapegraph-ai: asyncio.run() cannot be called from a running event loop

Hi there,

trying to get SmartScraperGraph running on Fast-API.

@app.post("/crawl")
async def crawl(request: Request):
    data = await request.json() 
    url = data.get('url')   

    try:  
        smart_scraper_graph =  SmartScraperGraph(
            prompt="List me all the articles",
            # also accepts a string with the already downloaded HTML code
            source=url,
            config=graph_config
        )

        result = smart_scraper_graph.run()

        print(result)

        # Access the URL field
        return result
    except Exception as e:
        print(f"Error in crawl: {e}")
        return None

Config

`graph_config = {
    "llm": {
         "model": "ollama/llama3",
        "temperature": 0,
        "format": "json",  # Ollama needs the format to be specified explicitly
        "base_url": "http://localhost:11434",  # set ollama URL arbitrarily
    },
    "embeddings": {
        "model": "ollama/nomic-embed-text",
        "base_url": "http://localhost:11434",  # set ollama URL arbitrarilyURL
    }
}`

Error:

Error in crawl: asyncio.run() cannot be called from a running event loop
/Users/konrad/Documents/Projects/product-spider/apps/service/main.py:171: RuntimeWarning: coroutine 'AsyncChromiumLoader.ascrape_playwright' was never awaited

Any idea? Thanks

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 6
  • Comments: 18 (1 by maintainers)

Most upvoted comments

Run this before the above code.

import nest_asyncio
nest_asyncio.apply()

This answer solved my problem.

Encountering this issue too, while trying to run the graph from an async function (in my case a NATS event handler), I found the following workaround.

Basically it executes the asyncio event loop on another thread, but waits for the executing in the current event loop.

import asyncio
from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor()

async def run_blocking_code_in_thread(blocking_func, *args, **kwargs):
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(executor, blocking_func, *args, **kwargs)
    
async def your_async_method():
    smart_scraper_graph = SmartScraperGraph(
        prompt=...,
        source=...,
        config=...
    )
    result = await run_blocking_code_in_thread(smart_scraper_graph.run)

Not sure if there are any downsides using this approach, as I am fairly new to working with Python event loops. Looking forward to built-in support

Hey there, yes Playwright uses asyncio under the hood so probably you are trying to run an asyncio routine inside another one (your async crawl method). Right now the .rnu method doeesn’t include a way to handle asynchronous calls but since it is a requested feature we will add it 😃

Also we will include other web driver like the one provided by Selenium

please add the all the code

Hey - I ran into smth similar while trying to wrap the smart scraper graph with some fastapi endpoints - what worked for me was to wrap the whole thing with run_in_threadpool from starlette.concurrency - running version 1.2.3