unit: Latest Docker Image Error Python failed to call 'asyncio.get_event_loop'

@tippexs Awesome, probably there would a bit of headache if not you 😉 Not a bad idea to clone repo in Dockerfile btw.

But to the point, I was able to build locally both unit:1.30.0-python3.10 using make build-python3.10 VERSION_python=3.10 and unit:1.30.0-python3.11 using make build-python3.11 VERSION_python=3.11 just as you described it.

Both are working alright with my setup where current official image nginx/unit:1.29.0-python3.11 fails.

Steps to reproduce results if anybody is concerned (python 3.10/3.11 fastapi asgi app) can be found here https://github.com/rafsaf/nginx_unit_issue_815

If there is anything I could do, don’t hesitate, glad I could help.

Hi,

Using the sample fastapi app (ASGI) in your documentation https://unit.nginx.org/howto/fastapi/ After updating to 1.29.0 with python 3.11 , yielded same error in this issue

2022/12/26 12:20:07 [info] 8422#8422 "fastapi" prototype started
2022/12/26 12:20:08 [info] 8427#8427 "fastapi" application started
2022/12/26 12:20:08 [alert] 0#8427 [unit] Python failed to call 'asyncio.get_event_loop'
2022/12/26 12:20:09 [notice] 8422#8422 app process 8427 exited with code 1
2022/12/26 12:20:09 [warn] 30882#30882 failed to start application "fastapi"
2022/12/26 12:20:09 [alert] 30882#30882 failed to apply new conf
2022/12/26 12:20:09 [notice] 30879#30879 process 8422 exited with code 0

using same sample application with python 3.10 module compiled to work on unit 1.29.0 is working without any issue

Thanks @0x6f677548 👍

Sorry I dropped the ball on the comms on this one. It was indeed fixed with this fix and shipped with 1.29.1.

Updates! I was able to make it work with Unit 1.29 and Python 3.10.7. As soon as I am trying to use the latest Python 3.10 version which is 3.10.9 the app starts to crash on boot.

To build a container with Unit 1.29 and Python 3.10.7 issue

git clone https://github.com/nginx/unit
cd unit
git checkout 1.29.0  # Optional; use to choose a specific Unit version
cd pkg/docker/
make build-python3.10 VERSION_python=3.10.7                                                                                                                                                                                                         

This will create a Docker image tagged as unit:1.29.0-python3.10. Feel free to test your apps with this image. In the meantime we are looking into the issue with Python Versions > 3.10.7

Here, perhaps this will help. This is the simplest recreation I can come up with. Create a Dockerfile with the following contents:

# Uncomment 1.28 to see working version
# FROM public.ecr.aws/nginx/unit:1.28.0-python3.10
FROM public.ecr.aws/nginx/unit:1.29.0-python3.11

WORKDIR /www
# port used by the listener in nginx-unit-config.json
EXPOSE 8080
# Copy our Nginx configuration file to the container
RUN echo '{"listeners":{"*:8080":{"pass":"applications/webapp"}},"applications":{"webapp":{"type":"python 3","path":"/www/","module":"nunittest.asgi"}}}' > /docker-entrypoint.d/config.json
# Copy our application code to the container
RUN mkdir /www/nunittest
RUN echo '' > /www/nunittest/__init__.py
RUN echo 'async def application(scope, receive, send):\n\
    await send({"type": "http.response.start", "status": 200})\n\
    \n\
    await send({"type": "http.response.body", "body": b"Hello, ASGI\\n"})\n' > /www/nunittest/asgi.py

Then run it with this command docker run --publish 8080:8080 --rm $(docker build -q .) which will build and run the dockerfile in the current directory.

You’ll see that with 1.29.0 the server crashes, but if you uncomment the 1.28.0 image and comment out 1.29.0 that the server launches without issue.

@rafsaf Thanks for taking the time and effort to test this!

I will look at getting the patches reviewed and merged.

@lcrilly sure thing I can do it, hope I can find time today and try it on evening with an app that broke after image upgrade

waiting patiently here for this bug to be fixed

@rafsaf we have a fix. Are you in a position to test it? https://github.com/ac000/unit/tree/asyncio

Sure, could help me out with an docker image or Dockerfile? I’d gladly will test it out.

diff --git a/src/python/nxt_python_asgi.c b/src/python/nxt_python_asgi.c
index 587a17cf..9fe5b56e 100644
--- a/src/python/nxt_python_asgi.c
+++ b/src/python/nxt_python_asgi.c

[…]

Please, split that into a refactoring patch that adds the function, and then one that does the fix, to keep my sanity when reviewing 😃

I have just tested Unit 1.29 using Python 3.10 and I get the same error as with Python 3.11.

However even Unit 1.28.0 with Python3.10 shows me

tippexs@beaker:/tmp$ docker run --publish 8080:8080 --rm $(docker build -q .)
/usr/local/bin/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, launching Unit daemon to perform initial configuration...
2023/01/13 13:33:40 [info] 10#10 unit 1.28.0 started
2023/01/13 13:33:40 [info] 12#12 discovery started
2023/01/13 13:33:40 [notice] 12#12 module: python 3.10.9 "/usr/lib/unit/modules/python3.unit.so"
2023/01/13 13:33:40 [info] 11#11 controller started
2023/01/13 13:33:40 [notice] 11#11 process 12 exited with code 0
2023/01/13 13:33:40 [info] 15#15 router started
2023/01/13 13:33:40 [info] 15#15 OpenSSL 1.1.1n  15 Mar 2022, 101010ef
{
        "certificates": {},
        "config": {
                "listeners": {},
                "applications": {}
        },

        "status": {
                "connections": {
                        "accepted": 0,
                        "active": 0,
                        "idle": 0,
                        "closed": 0
                },

                "requests": {
                        "total": 0
                },

                "applications": {}
        }
}
/usr/local/bin/docker-entrypoint.sh: Looking for certificate bundles in /docker-entrypoint.d/...
/usr/local/bin/docker-entrypoint.sh: Looking for configuration snippets in /docker-entrypoint.d/...
/usr/local/bin/docker-entrypoint.sh: Applying configuration /docker-entrypoint.d/config.json
2023/01/13 13:33:40 [info] 19#19 "webapp" prototype started
2023/01/13 13:33:40 [info] 20#20 "webapp" application started
2023/01/13 13:33:41 [alert] 0#20 [unit] Python failed to call 'asyncio.get_event_loop'
2023/01/13 13:33:41 [notice] 19#19 app process 20 exited with code 1
2023/01/13 13:33:41 [warn] 15#15 failed to start application "webapp"
2023/01/13 13:33:41 [alert] 15#15 failed to apply new conf
2023/01/13 13:33:41 [notice] 11#11 process 19 exited with code 0
/usr/local/bin/docker-entrypoint.sh: Error: HTTP response status code is '500'
{
        "error": "Failed to apply new configuration."
}

So for me 1.28 AND 1.29 are failing regardless of the Python Versions 3.10 or 3.11.

Using the Images from the AWS ECR Registry the Unit 1.28 Python 3.10 Demo Application is working.

We are looking into it. I need to check if the custom build for our images is working as we expect.

@tippexs hi, thank you for working on that case!

Just like temporary workaround would be great to have docker image with unit 1.29.0 and python 3.10

Hi Team - I am working with the Dev Team on an analysis about the root cause. Will let you all know once we have something to share.

Atm. it looks like it is something new in the combination of Unit 1.29 and Python 3.11.1 as Python 3.10 on 1.29 works just fine.

Trying to reproduce this locally (current unit master, Python 3.11).

With the above config I get

2022/12/21 15:13:58 [alert] 9398#9398 Python failed to import module "nunittest.asgi"
ModuleNotFoundError: No module named 'nunittest'

With the following

{
  "listeners": {
    "[::1]:8080": {
      "pass": "applications/webapp"
    }
  },

  "applications": {
    "webapp": {
      "type": "python",
      "path": "/srv/unit/python/app/",
      "module": "asgi"
    }
  }
}

$ cat /srv/unit/python/app/asgi.py

async def application(scope, receive, send):
    await send({"type": "http.response.start", "status": 200})

    await send({"type": "http.response.body", "body": b"Hello, ASGI\n"})

The app seems to start, but not really…

{
        "success": "Reconfiguration done."
}

From the log

2022/12/21 15:28:06 [info] 9642#9642 "webapp" application started
2022/12/21 15:28:06 [warn] 0#9642 [unit] Got invalid state transition on lifespan protocol
2022/12/21 15:28:06 [info] 0#9642 [unit] ASGI Lifespan processing exception
Traceback (most recent call last):
  File "/srv/unit/python/app/asgi.py", line 2, in application
    await send({"type": "http.response.start", "status": 200})
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Got invalid state transition on lifespan protocol

And when trying to access it

$ curl http://localhost:8080/
<!DOCTYPE html><title>Error 503</title><p>Error 503.

2022/12/21 15:28:45 [alert] 9642#9642 [unit] #8: Python failed to create 'client' pair
2022/12/21 15:28:45 [alert] 9642#9642 [unit] Python failed to call 'loop.call_soon'
ValueError: invalid literal for int() with base 10: ':1'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib64/python3.11/asyncio/base_events.py", line 762, in call_soon
    handle = self._call_soon(callback, args, context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/asyncio/base_events.py", line 778, in _call_soon
    handle = events.Handle(callback, args, self, context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SystemError: <class 'asyncio.events.Handle'> returned a result with an exception set