apprise-api: Worker timeout

❓ Question

Hi!

Thank you very much for both the project and API! I am using the docker image to post request to Apprise as a gateway to Gotify.

It works most of the time but at night I often miss some notifications. There are some logs but I don’t understand them: what triggers this worker timeout? The app calling apprise-api, or the gotify server that apprise-api calls?

[2020-07-28 22:42:27 +0000] [183] [CRITICAL] WORKER TIMEOUT (pid:201)
2020/07/28 22:42:31 [error] 198#198: *31 upstream prematurely closed connection while reading response header from upstream, client: 172.17.0.1, server: _, request: "POST /notify HTTP/1.1", upstream: "http://127.0.0.1:8080/notify", host: "localhost:8000"
2020/07/28 22:42:31 [warn] 198#198: *31 upstream server temporarily disabled while reading response header from upstream, client: 172.17.0.1, server: _, request: "POST /notify HTTP/1.1", upstream: "http://127.0.0.1:8080/notify", host: "localhost:8000"
172.17.0.1 - - [28/Jul/2020:22:42:32 +0000] "POST /notify HTTP/1.1" 502 494 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "-"
[2020-07-28 22:42:33 +0000] [210] [INFO] Booting worker with pid: 210

Logs since the start if needed (all the POST requests are coming from successful tests):

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.                                                                                                                                     
[s6-init] ensuring user provided files have correct perms...exited 0.                                                                                                                                             
[fix-attrs.d] applying ownership & permissions fixes...                                                                                                                                                           
[fix-attrs.d] done.                                                                                                                                                                                               
[cont-init.d] executing container initialization scripts...                                                                                                                                                       
[cont-init.d] done.                                                                                                                                                                                               
[services.d] starting services                                                                                                                                                                                    
[services.d] done.                                                                                                                                                                                                
Starting gunicorn                                                                                                                                                                                                 
Starting nginx                                                                                                                                                                                                    
2020/07/28 17:53:57 [notice] 197#197: using the "epoll" event method                                                                                                                                              
2020/07/28 17:53:57 [notice] 197#197: nginx/1.19.0
2020/07/28 17:53:57 [notice] 197#197: built by gcc 8.3.0 (Debian 8.3.0-6)
2020/07/28 17:53:57 [notice] 197#197: OS: Linux 4.9.0-12-amd64
2020/07/28 17:53:57 [notice] 197#197: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2020/07/28 17:53:57 [notice] 197#197: start worker processes
2020/07/28 17:53:57 [notice] 197#197: start worker process 198
[2020-07-28 17:53:58 +0000] [183] [INFO] Starting gunicorn 20.0.4
[2020-07-28 17:53:58 +0000] [183] [INFO] Listening at: http://0.0.0.0:8080 (183)
[2020-07-28 17:53:58 +0000] [183] [INFO] Using worker: sync
[2020-07-28 17:53:58 +0000] [201] [INFO] Booting worker with pid: 201
[2020-07-28 17:53:58 +0000] [202] [INFO] Booting worker with pid: 202
[2020-07-28 17:53:58 +0000] [203] [INFO] Booting worker with pid: 203
[2020-07-28 17:53:58 +0000] [204] [INFO] Booting worker with pid: 204
[2020-07-28 17:53:58 +0000] [205] [INFO] Booting worker with pid: 205
[2020-07-28 17:53:58 +0000] [206] [INFO] Booting worker with pid: 206
[2020-07-28 17:53:58 +0000] [207] [INFO] Booting worker with pid: 207
[2020-07-28 17:53:58 +0000] [208] [INFO] Booting worker with pid: 208
[2020-07-28 17:53:59 +0000] [209] [INFO] Booting worker with pid: 209
127.0.0.1 - - [28/Jul/2020:13:00:57 -0500] "GET / HTTP/1.0" 200 25873 "-" "curl/7.52.1"
172.17.0.1 - - [28/Jul/2020:18:00:57 +0000] "GET / HTTP/1.1" 200 25873 "-" "curl/7.52.1" "-"
2020/07/28 18:00:57 [info] 198#198: *1 client 172.17.0.1 closed keepalive connection
127.0.0.1 - - [28/Jul/2020:13:01:05 -0500] "POST /notify HTTP/1.0" 200 21 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
172.17.0.1 - - [28/Jul/2020:18:01:06 +0000] "POST /notify HTTP/1.1" 200 52 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "-"
127.0.0.1 - - [28/Jul/2020:13:01:14 -0500] "POST /notify HTTP/1.0" 200 21 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
172.17.0.1 - - [28/Jul/2020:18:01:14 +0000] "POST /notify HTTP/1.1" 200 52 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "-"
127.0.0.1 - - [28/Jul/2020:13:02:09 -0500] "POST /notify HTTP/1.0" 200 21 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
172.17.0.1 - - [28/Jul/2020:18:02:09 +0000] "POST /notify HTTP/1.1" 200 52 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "-"
127.0.0.1 - - [28/Jul/2020:13:02:12 -0500] "POST /notify HTTP/1.0" 200 21 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
172.17.0.1 - - [28/Jul/2020:18:02:12 +0000] "POST /notify HTTP/1.1" 200 52 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "-"
127.0.0.1 - - [28/Jul/2020:13:04:52 -0500] "POST /notify HTTP/1.0" 200 21 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
172.17.0.1 - - [28/Jul/2020:18:04:52 +0000] "POST /notify HTTP/1.1" 200 52 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "-"
127.0.0.1 - - [28/Jul/2020:13:04:57 -0500] "POST /notify HTTP/1.0" 200 21 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
172.17.0.1 - - [28/Jul/2020:18:04:57 +0000] "POST /notify HTTP/1.1" 200 52 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "-"
127.0.0.1 - - [28/Jul/2020:13:04:59 -0500] "POST /notify HTTP/1.0" 200 21 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
172.17.0.1 - - [28/Jul/2020:18:04:59 +0000] "POST /notify HTTP/1.1" 200 52 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "-"
127.0.0.1 - - [28/Jul/2020:13:05:11 -0500] "POST /notify HTTP/1.0" 200 21 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
172.17.0.1 - - [28/Jul/2020:18:05:11 +0000] "POST /notify HTTP/1.1" 200 52 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "-"
127.0.0.1 - - [28/Jul/2020:13:05:12 -0500] "POST /notify HTTP/1.0" 200 21 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
172.17.0.1 - - [28/Jul/2020:18:05:12 +0000] "POST /notify HTTP/1.1" 200 52 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "-"
127.0.0.1 - - [28/Jul/2020:13:05:13 -0500] "POST /notify HTTP/1.0" 200 21 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
172.17.0.1 - - [28/Jul/2020:18:05:13 +0000] "POST /notify HTTP/1.1" 200 52 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" "-"
127.0.0.1 - - [28/Jul/2020:13:05:15 -0500] "POST /notify HTTP/1.0" 200 21 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"

Can you help please? Thanks in advance!

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 20 (10 by maintainers)

Most upvoted comments

Hi again,

I don’t have the issue right now after several tests (and a docker network change). Just wanted to thank you again:

The latest issue on my side were due to docker not having external network access after a debian upgrade and nftables migration. I am not savvy enough to fix it properly so I passed the --network host parameter but hit another issue: I already have something on port 8080. So I was about to build a new docker without nginx and only gunicorn. And then I saw you already provide such an image ❤️.

nicofrand on Aug 26, 2020

Maybe a solution would be to allow to configure the timeout, so I can increase it myself without breaking the feature for other users?

I love this idea; i think what I’ll do is just bump the 2.5 seconds to 4.0 seconds (just a wee bit more then it is and may fix your issue right out of the box).

~~But (taking your advice) I’ll also add the variables read_timeout= and connect_timeout accessible from all Apprise URLs that deal with web/api query based services too (for consistency).~~

Edit:

cto stands for Connect Timeout (defaults to 4.0 seconds)
rto stands for Read Timeout (and now also defaults to 4.0 seconds)

You can over-ride both and/or just one of them; so basically your Apprise URL will look like (examples):

gotify://url/details/here?rto=15
gotify://url/details/here?cto=15
gotify://url/details/here?cto=15&rto=30

caronc on Aug 1, 2020