cvat: DEXTR - Semi-Automatic Annotation - 504 Gateway Timeout

My actions before raising this issue

• Read/searched the docs
• Searched past issues

I have searched the docs and past issues - could not figure it out, kindly help.

Expected Behaviour

Automatic instance segmentation annotation

Current Behaviour

Getting below error:

Gateway Timeout
The gateway did not receive a timely response from the upstream server or application.

Possible Solution

From logs, it looks like connection not reaching the docker instance - may be providing some advice on configuring proxy/firewall will help us to understand the issue better.

Steps to Reproduce (for bugs)

Followed the steps provided in the installation guide:

https://openvinotoolkit.github.io/cvat/docs/for-users/installation/

Note: Installation is successful, I will share the logs and details below

Context

When trying to use AI Tool getting timeout error

Your Environment

• Git hash commit (git log -1):

commit 3afd7e827878112e6d6bf9a435e7b96d38baaa26 (HEAD -> develop, origin/develop, origin/HEAD)
Author: Dmitry Kruchinin <33020454+dvkruchinin@users.noreply.github.com>
Date:   Mon Jun 7 11:59:10 2021 +0300
    Cypress test. Create an object, save, undo, save, redo, save. (#3299)

• Docker version docker version (e.g. Docker 17.0.05):

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 12:00:45 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:58:56 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

• Are you using Docker Swarm or Kubernetes? No

• Operating System and version (e.g. Linux, Windows, MacOS):

NAME="Ubuntu"
VERSION="21.04 (Hirsute Hippo)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 21.04"
VERSION_ID="21.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=hirsute
UBUNTU_CODENAME=hirsute

• Code example or link to GitHub repo or gist to reproduce problem: Same as official repo

• Other logs:

docker ps:

CONTAINER ID   IMAGE                                   COMMAND                  CREATED        STATUS                  PORTS                                                                                
                             NAMES
3e43c3e7b553   alpine:3.11                             "/bin/sh -c '/bin/sl…"   5 hours ago    Up 5 hours                                                                                                   
                             nuclio-local-storage-reader
296d755cc856   cvat/openvino.dextr:latest              "processor"              11 hours ago   Up 11 hours (healthy)   0.0.0.0:49153->8080/tcp, :::49153->8080/tcp                                          
                             nuclio-nuclio-openvino-dextr
8e4e3f6f04ae   nginx:stable-alpine                     "/docker-entrypoint.…"   11 hours ago   Up 11 hours             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:808
0->80/tcp, :::8080->80/tcp   cvat_proxy
f1ec02907177   openvino/cvat_ui                        "/docker-entrypoint.…"   11 hours ago   Up 11 hours             80/tcp                                                                               
                             cvat_ui
1b32376d2c98   openvino/cvat_server                    "/usr/bin/supervisord"   11 hours ago   Up 11 hours             8080/tcp                                                                             
                             cvat
19675add9961   postgres:10-alpine                      "docker-entrypoint.s…"   11 hours ago   Up 11 hours             5432/tcp                                                                             
                             cvat_db
f52bcf45003b   redis:4.0-alpine                        "docker-entrypoint.s…"   11 hours ago   Up 11 hours             6379/tcp                                                                             
                             cvat_redis
0090c6624c5d   quay.io/nuclio/dashboard:1.5.16-amd64   "/docker-entrypoint.…"   11 hours ago   Up 11 hours (healthy)   80/tcp, 0.0.0.0:8070->8070/tcp, :::8070->8070/tcp                                    
                             nuclio

docker logs cvat: (only error part)

[Mon Jun 07 18:55:17.009057 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:34580] [2021-06-07 18:55:17,008] INFO cvat.server.task_2: get repository request
2021-06-07 18:55:17,010 DEBG 'runserver' stderr output:
[Mon Jun 07 18:55:17.010097 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:34580] INFO - 2021-06-07 18:55:17,008 - views - get repository request
2021-06-07 18:55:20,822 DEBG 'runserver' stderr output:
[Mon Jun 07 18:55:20.822152 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:34608] INFO - 2021-06-07 18:55:20,822 - views - {"client_id":582112,"name":"Send user activity","time":"2021
-06-07T18:55:20.723000Z","payload":{"working_time":50033},"is_active":true,"username":"admin"}
2021-06-07 18:56:35,536 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:35.536331 2021] [wsgi:error] [pid 322:tid 140652568270592] [client 172.28.0.7:34694] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/a
pi, referer: https://my-server-ip/tasks/2/jobs/2
2021-06-07 18:56:35,637 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:35.637400 2021] [wsgi:error] [pid 255:tid 140652446222080] [remote 172.28.0.7:34694] [2021-06-07 18:56:35,637] ERROR django.request: Gateway Timeout: /api/v1/lambda/functions/openvino-de
xtr
2021-06-07 18:56:35,637 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:35.637517 2021] [wsgi:error] [pid 255:tid 140652446222080] [remote 172.28.0.7:34694] ERROR - 2021-06-07 18:56:35,637 - log - Gateway Timeout: /api/v1/lambda/functions/openvino-dextr
2021-06-07 18:56:51,575 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:51.575198 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:35000] INFO - 2021-06-07 18:56:51,575 - views - {"job_id":2,"task_id":2,"client_id":582112,"name":"Load job"
,"time":"2021-06-07T18:55:20.723000Z","payload":{"duration":1450,"frame count":7,"track count":0,"object count":0,"box count":0,"polygon count":0,"polyline count":0,"points count":0,"cuboids count":0,"tag
 count":0},"is_active":true,"username":"admin"}
2021-06-07 18:56:51,575 DEBG 'runserver' stderr output:
[Mon Jun 07 18:56:51.575359 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:35000] INFO - 2021-06-07 18:56:51,575 - views - {"job_id":2,"task_id":2,"client_id":582112,"name":"Fit image
","time":"2021-06-07T18:55:22.176000Z","payload":{},"is_active":true,"username":"admin"}
[Mon Jun 07 18:56:51.575465 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:35000] INFO - 2021-06-07 18:56:51,575 - views - {"client_id":582112,"name":"Send user activity","time":"2021
-06-07T18:56:51.481000Z","payload":{"working_time":14505},"is_active":true,"username":"admin"}
2021-06-07 18:57:30,995 DEBG 'runserver' stderr output:
[Mon Jun 07 18:57:30.995435 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:35178] [2021-06-07 18:57:30,995] INFO cvat.server.task_2: get repository request
2021-06-07 18:57:30,995 DEBG 'runserver' stderr output:
[Mon Jun 07 18:57:30.995579 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:35178] INFO - 2021-06-07 18:57:30,995 - views - get repository request
2021-06-07 18:58:41,613 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-06-07 18:58:41,612 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
2021-06-07 18:58:41,812 DEBG 'rqworker_default_1' stderr output:
DEBUG - 2021-06-07 18:58:41,812 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
2021-06-07 18:59:59,108 DEBG 'runserver' stderr output:
[Mon Jun 07 18:59:59.108022 2021] [wsgi:error] [pid 255:tid 140652454614784] [remote 172.28.0.7:35766] INFO - 2021-06-07 18:59:59,107 - views - {"client_id":582112,"name":"Send user activity","time":"2021
-06-07T18:59:59.009000Z","payload":{"working_time":50882},"is_active":true,"username":"admin"}
2021-06-07 19:01:13,421 DEBG 'runserver' stderr output:
[Mon Jun 07 19:01:13.421313 2021] [wsgi:error] [pid 322:tid 140652568004352] [client 172.28.0.7:35852] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/a
pi, referer: https://my-server-ip/tasks/2/jobs/2
2021-06-07 19:01:13,590 DEBG 'runserver' stderr output:
[Mon Jun 07 19:01:13.590114 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:35852] [2021-06-07 19:01:13,590] ERROR django.request: Gateway Timeout: /api/v1/lambda/functions/openvino-de
xtr
2021-06-07 19:01:13,590 DEBG 'runserver' stderr output:
[Mon Jun 07 19:01:13.590248 2021] [wsgi:error] [pid 255:tid 140652437829376] [remote 172.28.0.7:35852] ERROR - 2021-06-07 19:01:13,590 - log - Gateway Timeout: /api/v1/lambda/functions/openvino-dextr
2021-06-07 19:01:33,901 DEBG 'runserver' stderr output:
[Mon Jun 07 19:01:33.901047 2021] [core:info] [pid 322:tid 140652568803072] [client 172.28.0.7:36202] AH00128: File does not exist: /home/django/static/rest_framework/css/bootstrap.min.css.map
2021-06-07 19:01:54,672 DEBG 'rqworker_default_0' stderr output:
DEBUG - 2021-06-07 19:01:54,671 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

nuctl get function:

  NAMESPACE |      NAME      | PROJECT | STATE | NODE PORT | REPLICAS  
  nuclio    | openvino-dextr | cvat    | ready |     49153 | 1/1  

nuctl version:

Client version:
"Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"root@gc-ase-ml-data-annotate1:/data/cvat# 

systemctl show --property=Environment docker:

Environment=HTTP_PROXY=http://my-proxy-ip:8080 HTTPS_PROXY=http://my-proxy-ip:8080 NO_PROXY=localhost,127.0.0.1

cat ~/.docker/config.json:

{
 "proxies":
 {
   "default":
   {
     "httpProxy": "http://my-proxy-ip:8080",
     "httpsProxy": "http://my-proxy-ip:8080",
     "noProxy": "*.mydomain.com,localhost,127.0.0.0/8"
   }
 }
}

docker-compose.override.yml:

version: '3.3'
services:
  cvat_proxy:
    environment:
      CVAT_HOST: 'my-host-ip'
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - /etc/ssl/certs:/etc/ssl/certs
  cvat:
    environment:
      ALLOWED_HOSTS: '*'

Screenshots:

API Page:

API Output:

Logs for this request:

Next steps

I have joined gitter and shared details - waiting for response.