unit: nginx unit fails when web application load time is longer than 60 seconds

I have a python web application, that has a very long loading time due to some heavy database calculations that need to be run one on start time of the application. When running the application in a docker container with nginx unit, the setup of the application fails. The curl request in the docker entrypoint that configures the web app (https://github.com/nginx/unit/blob/master/pkg/docker/docker-entrypoint.sh#L43) fails with the following error when the application takes longer than 60 seconds to start:

curl: (52) Empty reply from server

Here is my sample application code:

Directory Structure:

- app.py
- config.json
- Dockerfile
- requirements.txt

Dockerfile:

FROM nginx/unit:1.26.1-python3.9

COPY requirements.txt /config/requirements.txt
RUN pip install -r /config/requirements.txt 

COPY config.json /docker-entrypoint.d/config.json
COPY app.py /www/app.py

app.py:

from flask import Flask
import time

app = Flask(__name__)
application = app

# Artificially ensure a high load time
time.sleep(70)

@app.route('/')
def hello_world():
    return 'Hello, World!'

config.json:

{
    "settings": {
        "http":{
            "header_read_timeout": 300,
            "body_read_timeout":300,
            "idle_timeout":300,
            "send_timeout":300        
       }
    },
    "listeners": {
        "*:8080": {
            "pass": "applications/python_app"
        }
    },
    "applications": {
        "python_app": {
            "type": "python",
            "path": "/www/",
            "module": "app",
            "callable": "app",
            "limits":{
                "timeout": 300,
                "requests": 500
            }
        }
    }
}

requirements.txt:

Flask==2.0.2

I tried to configure all timeouts in a way that the should not produce problems, but this did not help. Is there anything that I am doing wrong?

Thanks for any support in advance already!

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 1
Comments: 16 (9 by maintainers)

Commits related to this issue

Increased readtimeout for configuration endpoint. Closes: <https://github.com/nginx/unit/issues/676> — committed to nginx/unit by tippexs 2 years ago

Most upvoted comments

Hi @Phylu we have scheduled the next release for end of May (Thu. May 26th). So it will be around the corner.

tippexs on May 12, 2022

Hi @Phylu

sorry for the really long delay with this issue. I was finally able to reproduce it constantly and found the timeout in the unit source code. We are closing the connection to the config http endpoint after 60 seconds.

We will check if it makes sense to make this timeout configurable. For now and your issue it is faster to quickly change the sources. file: nxt_controller.c - nxt_controller_conn_read_state

diff -r 1a08f884b24e src/nxt_controller.c
--- a/src/nxt_controller.c      Thu Dec 02 18:23:00 2021 +0300
+++ b/src/nxt_controller.c      Sun Apr 24 15:09:02 2022 -0400
@@ -725,7 +725,7 @@
 
     .timer_handler = nxt_controller_conn_read_timeout,
     .timer_value = nxt_controller_conn_timeout_value,
-    .timer_data = 60 * 1000,
+    .timer_data = 90 * 1000,
 };

You can change the timeout from 60 seconds to 90 or even 120 seconds. To make the new unitd binary work with the docker container you have to use the version 1.26.1 of the unit source and configure unit from source using the following configure command.

hg clone http://hg.nginx.org/unit -r 1.26.1

./configure --prefix=/usr --state=/var/lib/unit --control=unix:/var/run/control.unit.sock --pid=/var/run/unit.pid --log=/var/log/unit.log --tmp=/var/tmp --user=unit --group=unit --openssl --libdir=/usr/lib/x86_64-linux-gnu --cc-opt='-g -O2 -ffile-prefix-map=/unit=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --ld-opt='-specs=/usr/share/dpkg/no-pie-link.specs -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie' --modules=/usr/lib/unit/modules

type make to build unit. Copy the new unitd binary into the container: Copy the unitd in the build context of docker by issuing cp unit/src/build/unitd . and add this to your Dockerfile

COPY unitd /usr/sbin/unitd

Let me know if you need anything else or in case you have more questions.

tippexs on Apr 24, 2022

@Phylu I have something to test. The issue may not be in the Unit source. We were not able to reproduce the issue on other OSes like Ubuntu, Mac or CentOS.

Will share some idea how to fix this later today.

Update - Still working on it. libcurl does not have a read timeout. That said, curl would technically wait forever for a response from the server.

tippexs on Apr 24, 2022

@hongzhidao Here is the debug log from running it in docker via CMD ["unitd-debug","--no-daemon","--control","unix:/var/run/control.unit.sock"] and setting the time.sleep to 120 seconds. I am currently on another machine and the timing here seems a little bit different and this ensures that it fails.

debug.log

You won’t see the curl error message though as the curl statement in the entrypoint contains the silent flag. If you overwrite the entrypoint and remove the -s there, you will see the following curl error at the end of the logs:

2022/04/15 09:46:58.457 [debug] 16#16 epoll_wait(3) timeout:-1
100   586    0     0  100   586      0      9  0:01:05  0:01:00  0:00:05     0
curl: (52) Empty reply from server

Phylu on Apr 15, 2022

Perfect. That is the exact same approach I am following right now. Let me quickly test this with and without docker and share the result here.

tippexs on Apr 14, 2022