cvat: Automatic annotation 500 ServerError

here is my solution, see if it helps for you. Updating the function.yaml like below

Wrap solution up:

1. Modify functions yaml as @yeahydq did above: https://user-images.githubusercontent.com/18577463/152193385-f4e661ec-a617-44b5-a6f3-1747a7a62f35.png
2. then just run: sudo ufw allow proto tcp from any to 172.17.0.1 as @KiirkH mentioned above. Yes, a security issue could rise from this.

These steps worked for me.

Thanks!

Faced same issue recently. After digging around investigated that processors are deployed not to the cvat_cvat network (where main ‘staff’ is located) and UFW ‘kindly’ blocked all requests from nuclio container to them. My dirty hotfix with room for security improvement is below:

# Add docker0 network adapter address to the UFW exceptions
$ sudo ufw allow proto tcp from any to 172.17.0.1

This is still an issue. Is there any way to use an older release in which automatic annotation is working?

Any solution for this yet?

@EldarKurbanov , I will look at the issue today. Sorry for the delay with my reply.

Another thing you can do is to change the function.yaml files to explicitely use the correct network. In this case cvat_cvat:

Change the following code:

  platform:
    attributes:
      restartPolicy:
        name: always
        maximumRetryCount: 3
      mountMode: volume

to

  platform:
    attributes:
      restartPolicy:
        name: always
        maximumRetryCount: 3
      mountMode: volume
      network: cvat_cvat

After this i could invoke the function correctly from the dashboard. But from a cvat task I still got the same 500 error as before. That because using the nuclio like cvat does somehow still leads to the post request being send to 172.17.0.1.

I am not sure why. I will further debug this and add logs to this thread.

I’m experiencing the same problem. I’ve deployed some functions using nuctl as described in the docs.

nuctl get function reports function status as ready:

  NAMESPACE |               NAME               | PROJECT | STATE | NODE PORT | REPLICAS  
  nuclio    | openvino.dextr                   | cvat    | ready |     42503 | 1/1       
  nuclio    | openvino.omz.public.yolo-v3-tf   | cvat    | ready |     36835 | 1/1       
  nuclio    | tf.faster_rcnn_inception_v2_coco | cvat    | ready |     35819 | 1/1 

When I try auto annotating a task via CVAT dashboard, This error appears:

Fetching inference status for the task 1
Error: Inference status for the task 1 is failed. requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations

I get this log in nuclio container:

20.09.26 11:20:09.356 dashboard.platform.docker (D) Container is healthy {"containerID": "4b7f5cc22a8f00b4d81fe2e6bd5dc60a46d6bda8e02b8c0ff0d807fe743952b1"}
20.09.26 11:20:11.133 .api/function_invocations (W) Failed to invoke function {"err": "Failed to send HTTP request", "errVerbose": "\nError - Post \"http://172.17.0.1:35819\": dial tcp 172.17.0.1:35819: i/o timeout\n    /nuclio/pkg/platform/abstract/invoker.go:119\n\nCall stack:\nFailed to send HTTP request\n    /nuclio/pkg/platform/abstract/invoker.go:119\nFailed to send HTTP request", "errCauses": [{"error": "Post \"http://172.17.0.1:35819\": dial tcp 172.17.0.1:35819: i/o timeout"}]}
20.09.26 11:20:11.133          dashboard.server (D) Handled request {"requestMethod": "POST", "requestPath": "/api/function_invocations", "requestHeaders": {"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"],"Connection":["close"],"Content-Length":["177225"],"Content-Type":["application/json"],"User-Agent":["python-requests/2.24.0"],"X-Nuclio-Function-Name":["tf.faster_rcnn_inception_v2_coco"],"X-Nuclio-Function-Namespace":["nuclio"],"X-Nuclio-Log-Level":[""],"X-Nuclio-Path":["/"],"X-Nuclio-Project-Name":["cvat"],"X-Nuclio-Target":["tf.faster_rcnn_inception_v2_coco"]}, "requestBody": "{\"image\": \"iVBOgoA..........4AkJggg==\"}", "responseStatus": 500, "responseBody": "{\"error\": \"Failed to invoke function\"}", "responseTime": "30.174545501s"}

cvat container logs:

2020-09-26 13:09:37,674 DEBG 'rqworker_low' stderr output:
DEBUG:rq.worker:Handling failed execution of job 1ed2e996-87fc-4a8f-8445-0e0a61aef933

2020-09-26 13:09:37,676 DEBG 'rqworker_low' stderr output:
ERROR:rq.worker:Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.5/dist-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.5/dist-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 473, in __call__
    kwargs.get("threshold"), kwargs.get("mapping"))
  File "/home/django/cvat/apps/lambda_manager/views.py", line 342, in _call_detector
    "threshold": threshold})
  File "/home/django/cvat/apps/lambda_manager/views.py", line 175, in invoke
    response = self.gateway.invoke(self, payload)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 81, in invoke
    'x-nuclio-path': '/'
  File "/home/django/cvat/apps/lambda_manager/views.py", line 56, in _http
    reply.raise_for_status()
  File "/usr/local/lib/python3.5/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.5/dist-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.5/dist-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 473, in __call__
    kwargs.get("threshold"), kwargs.get("mapping"))
  File "/home/django/cvat/apps/lambda_manager/views.py", line 342, in _call_detector
    "threshold": threshold})
  File "/home/django/cvat/apps/lambda_manager/views.py", line 175, in invoke
    response = self.gateway.invoke(self, payload)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 81, in invoke
    'x-nuclio-path': '/'
  File "/home/django/cvat/apps/lambda_manager/views.py", line 56, in _http
    reply.raise_for_status()
  File "/usr/local/lib/python3.5/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations

2020-09-26 13:09:37,677 DEBG 'rqworker_low' stderr output:
DEBUG:rq.worker:Invoking exception handler <function rq_handler at 0x7f3ddca79e18>

Also there is no log for nuclio-nuclio-tf.faster_rcnn_inception_v2_coco container during the process.

All three functions fail to be invoked and I get same error message.

My CVAT env details:

Server version: 1.1
Core version: 3.5.0
Canvas version: 2.0.2
UI version: 1.8.4