playwright-python: "RuntimeError: This event loop is already running" When attempting REPL example

@mxschmitt thanks for the quick response! nest_asyncio isn’t a standard package or one installed as a requirement for playwright. As such, I’m guessing most users wouldn’t have it installed. However, that did solve the problem:

In [1]: import nest_asyncio                                                                                                                                   

In [2]: nest_asyncio.apply()                                                                                                                                  

In [3]: from playwright import sync_playwright                                                                                                                

In [4]: playwright = sync_playwright().start()                                                                                                                

In [5]:     

My use-case within IPython is largely API exploration and interactive process development before writing a script or integration with an application.

Additional note, your example works perfectly in the base python REPL.

It’s been almost two years and this still isn’t fixed that I can find - playwright is explicitly stopping me from using nested loops:

import nest_asyncio
nest_asyncio.apply()
​
from playwright.sync_api import sync_playwright
playwright = sync_playwright().start()
---------------------------------------------------------------------------
Error                                     Traceback (most recent call last)
Input In [3], in <cell line: 2>()
      1 from playwright.sync_api import sync_playwright
----> 2 playwright = sync_playwright().start()

File c:\users\bubth\development\soc2\venv\lib\site-packages\playwright\sync_api\_context_manager.py:76, in PlaywrightContextManager.start(self)
     75 def start(self) -> SyncPlaywright:
---> 76     return self.__enter__()

File c:\users\bubth\development\soc2\venv\lib\site-packages\playwright\sync_api\_context_manager.py:42, in PlaywrightContextManager.__enter__(self)
     40             own_loop = loop
     41         if loop.is_running():
---> 42             raise Error(
     43                 """It looks like you are using Playwright Sync API inside the asyncio loop.
     44 Please use the Async API instead."""
     45             )
     47         def greenlet_main() -> None:
     48             loop.run_until_complete(self._connection.run_as_sync())

Error: It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead.

The workarounds don’t work for those of us using windows, and they don’t appear to work in WSL either - is there a reason we can’t get more information on why this is still closed two years later with no traction and no functional workarounds for presumably a fairly normal use case of using a jupyter notebook for exploratory programming?

It seems hard to believe that we just assume everyone will use raw iPython for everything so I’m confused by the reluctance to actually fix this.

If I’m understanding the asyncio base_events.py error, it’s because there’s no implementation of the _make_subprocess_transport function - is that not something that would be on the playwright package to implement in it’s async implementation?

If I get time tonight ill play around with it- I dont see any reason for it not to run in the existing loop - or are there architectural challenges with it sharing the loop?

On Tue, Oct 6, 2020, 8:35 AM Pavel Feldman notifications@github.com wrote:

My understanding is that resolving this requires cooperation between the embedder that controls the loop and sync Playwright that also wants to dispatch its messages forever. And I don’t think such cooperation is the right way to go. Sync API is designed as self-contained, assumes it controls execution. Is there a way to use async in Jupyter?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/microsoft/playwright-python/issues/178#issuecomment-704312511, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC5CHIMV52GR76C7R7XFYKTSJMTMLANCNFSM4QLBAFHQ .

@ddelange Thanks. I found a workaround on windows now. NOT RECOMMEND!

From https://github.com/jupyter/notebook/issues/5916, https://github.com/minrk/ipykernel/commit/079f072a8e90422dc74270992589c56ad9f7f9f2, https://stackoverflow.com/questions/44633458/why-am-i-getting-notimplementederror-with-async-and-await-on-windows/74311290#74311290, https://github.com/ipython/ipykernel/blob/99a1becaa958b33d80fe337fdbc41305030fdb6d/ipykernel/kernelapp.py#L633-L637. I found a way to make it work hackly (not recommend) on windows.

Comment asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy()) in %LOCALAPPDATA%\Programs\Python\Python310\Lib\site-packages\ipykernel\kernelapp.py (change to your path).

Then the following code will work in jupyter notebook.

import asyncio
from playwright.async_api import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(channel="chrome", headless=False, slow_mo=100)
        page = await browser.new_page()
        await page.goto("https://cnki.net/")
        await browser. Close()

await main()

for iPython / Jupiter notebooks it seems like the current workaround would be to use that before you run your shell:

import nest_asyncio
nest_asyncio.apply()

Out of curiosity to understand more the user case. You are trying out Playwright in a REPL or are you using it only with with the REPL in the end?

still does not work in jupyter (vscode interactive).

Sync API is not allowed!

import asyncio
from playwright.async_api import async_playwright
from playwright.sync_api import sync_playwright

# https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269
import nest_asyncio
nest_asyncio.apply()
# Error: It looks like you are using Playwright Sync API inside the asyncio loop.
# Please use the Async API instead.
p = sync_playwright().start()

Async API is not worked!

import asyncio
from playwright.async_api import async_playwright
from playwright.sync_api import sync_playwright

# https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269
import nest_asyncio
nest_asyncio.apply()

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(channel="chrome",headless=False, slow_mo=100)
        page = await browser.new_page()
        await page.goto("https://cnki.net/")
        await browser.close()
# NotImplementedError
asyncio.run(main())

Maybe I should switch to selenium, see run-selenium-in-jupyter-notebook-on-wsl2-or-ubuntu。

I have another question, why can’t I use sync API in jupyter? I have to write a lot await in async API. The sync API is simple and straight, performance or jupyter does not allowed?

still does not work in jupyter (vscode interactive).

Sync API is not allowed!

import asyncio
from playwright.async_api import async_playwright
from playwright.sync_api import sync_playwright

# https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269
import nest_asyncio
nest_asyncio.apply()
# Error: It looks like you are using Playwright Sync API inside the asyncio loop.
# Please use the Async API instead.
p = sync_playwright().start()

Async API is not worked!

import asyncio
from playwright.async_api import async_playwright
from playwright.sync_api import sync_playwright

# https://github.com/microsoft/playwright-python/issues/178#issuecomment-680249269
import nest_asyncio
nest_asyncio.apply()

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(channel="chrome",headless=False, slow_mo=100)
        page = await browser.new_page()
        await page.goto("https://cnki.net/")
        await browser.close()
# NotImplementedError
asyncio.run(main())

Maybe I should switch to selenium, see run-selenium-in-jupyter-notebook-on-wsl2-or-ubuntu。

If you want to use an async framework (playwright-python is natively async) inside Jupyter Notebooks, the ipykernel maintainer suggests to use the async api directly (await & async def syntax), instead of the sync wrappers. They don’t support sync wrappers around async libs…

@mxschmitt not sure why its closed - this problem still exists. You can reproduce this on the latest versions of jupyter+playwright

Found a workaround for getting Playwright working within Jupyter.

It leverages an awesome python library called RPyc to run Playwright outside of the kernel while keeping the API the same.

!pip install rpyc

import rpyc
import subprocess
subprocess.Popen('rpyc_classic.py -m forking'.split())  # start RPyc worker

conn = rpyc.classic.connect('localhost')
rplaywright = conn.modules['playwright']  

# rplaywright has the same API as playwright
pr = rplaywright.sync_playwright().start()
browser = pr.chromium.launch()
page = browser.newPage()
page.goto('http://whatsmyuseragent.org/')
page.screenshot(path='example.png')
browser.close()

# Grab screen
from PIL import Image
from io import BytesIO

browser = pr.chromium.launch()

page = browser.newPage()
page.goto('http://localhost:4000/v/tf3nhh')
img=page.screenshot(path='example.png')
Image.open(BytesIO(img))

Currently, playwright disallows attaching to a running loop: https://github.com/microsoft/playwright-python/blob/d98d38b89b4864966218b1333c9399fc35487725/playwright/main.py#L79-L83

This should be fine in plain IPython (does not launch an asyncio event loop in the main thread), but will fail in Jupyter that uses tornado>=5 (launches asyncio event loop on startup under the hood).

As calling run_until_complete is disallowed on running loops, this will fail in Jupyter.

Minimal reproducible example that will run in latest IPython, but will break on Jupyter environments that use tornado>=5 under the hood:

import asyncio
print(asyncio.run(asyncio.sleep(0)))

So for Jupyter, additionally a pip install 'tornado<5' 'notebook<5.7.9' should fix loop issues introduced in tornado>=5 (and if you need jupyterhub, use jupyterhub<0.9.0). refs https://github.com/jupyter/notebook/issues/3397#issuecomment-675566100 https://github.com/ipython/ipykernel/issues/548