connaisseur: "RuntimeError: Event loop is closed" error found in connaissuer

Describe the bug We deploy connaisseur in k8s cluster as a webhook to validate container images. We occasionally run into connaisseur-webhook timeout problem and find following exception around the same time timeout is observed.

 Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x7f224ca5a3e0>

Traceback (most recent call last):
   File "/usr/local/lib/python3.11/asyncio/base_subprocess.py", line 126, in __del__
     self.close()
   File "/usr/local/lib/python3.11/asyncio/base_subprocess.py", line 104, in close
     proto.pipe.close()
  File "/usr/local/lib/python3.11/asyncio/unix_events.py", line 765, in close

    self.write_eof()
       File "/usr/local/lib/python3.11/asyncio/unix_events.py", line 751, in write_eof

    self._loop.call_soon(self._call_connection_lost, None)
      File "/usr/local/lib/python3.11/asyncio/base_events.py", line 761, in call_soon

    self._check_closed()

  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed

    raise RuntimeError('Event loop is closed')

Is this error causing connaisseur non-responding?

Optional: Versions (please complete the following information as relevant):

  • OS: <OS and version>
  • Kubernetes Cluster: 1.26
  • Notary Server: <notary and version>
  • Container registry: <container registry and version>
  • Connaisseur: 3.0.0
  • Other: <other relevant tools and versions>

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 20 (10 by maintainers)

Most upvoted comments

Ok, that indeed looks like a regular timeout. Since the k8s API allows for a maximum of 30s during admission for the admission controllers like Connaisseur to respond, we explicitly timeout after ~29s to be able to deliver this response.

From the traceback we see that the cancel happended while the Cosign command was running. We’re internally calling the Cosign binary as SIGSTORE_NO_CACHE=1 cosign verify --output text ${IMAGE} and a bunch of other options depending on configuration. Since that part is outside our code base, we do have limited insight. If Connaisseur has log level debug set, we provide the --verbose flag to the Cosign binary as well, which should help with debugging (though it might be that these logs are lost in case of a timeout as the handling function is cancelled 😅). What you can do otherwise is to run the Cosign binary as a standalone program for your image to benchmark with debugging enabled. If the issue is persistent for the image, then this should shed some light on the root cause.