connaisseur: "RuntimeError: Event loop is closed" error found in connaissuer
Describe the bug We deploy connaisseur in k8s cluster as a webhook to validate container images. We occasionally run into connaisseur-webhook timeout problem and find following exception around the same time timeout is observed.
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x7f224ca5a3e0>
Traceback (most recent call last):
File "/usr/local/lib/python3.11/asyncio/base_subprocess.py", line 126, in __del__
self.close()
File "/usr/local/lib/python3.11/asyncio/base_subprocess.py", line 104, in close
proto.pipe.close()
File "/usr/local/lib/python3.11/asyncio/unix_events.py", line 765, in close
self.write_eof()
File "/usr/local/lib/python3.11/asyncio/unix_events.py", line 751, in write_eof
self._loop.call_soon(self._call_connection_lost, None)
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 761, in call_soon
self._check_closed()
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
raise RuntimeError('Event loop is closed')
Is this error causing connaisseur non-responding?
Optional: Versions (please complete the following information as relevant):
- OS: <OS and version>
- Kubernetes Cluster: 1.26
- Notary Server: <notary and version>
- Container registry: <container registry and version>
- Connaisseur: 3.0.0
- Other: <other relevant tools and versions>
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 20 (10 by maintainers)
Ok, that indeed looks like a regular timeout. Since the k8s API allows for a maximum of 30s during admission for the admission controllers like Connaisseur to respond, we explicitly timeout after ~29s to be able to deliver this response.
From the traceback we see that the cancel happended while the Cosign command was running. We’re internally calling the Cosign binary as
SIGSTORE_NO_CACHE=1 cosign verify --output text ${IMAGE}and a bunch of other options depending on configuration. Since that part is outside our code base, we do have limited insight. If Connaisseur has log level debug set, we provide the--verboseflag to the Cosign binary as well, which should help with debugging (though it might be that these logs are lost in case of a timeout as the handling function is cancelled 😅). What you can do otherwise is to run the Cosign binary as a standalone program for your image to benchmark with debugging enabled. If the issue is persistent for the image, then this should shed some light on the root cause.