gatekeeper: TLS handshake error from: EOF

Not stale.

We’re testing today and I will report back soon!

still an open issue

https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/364 resolved this. I can confirm no longer see the issue in AKS K8S after the AKS master nodes and kube-system pods are upgraded to 1.26.

Same here on eks 1.24 and gatekeeper 3.11.1. 2023/06/14 13:07:49 http: TLS handshake error from xxxxxxxxxxx: EOF ││ 2023/06/14 13:07:49 http: TLS handshake error from xxxxxxxxxxx EOF ││ 2023/06/14 13:07:49 http: TLS handshake error from xxxxxxxxxxx: EOF

Any solutions recommended ?

Seeing this error on 1.27.1 with gatekeeper v3.11.0 not sure if this is causing issues with timeouts for leaderelection or not

We’re currently running version 1.25.15 of kube and running version v3.12.0 of the opa gatekeeper and still seeing this error.

Still happening on AKS 1.25.5 and gatekeeper v3.11.1 Anyone found something on this?

Same here on GKE 1.25.7 with managed ASM

We’re testing today and I will report back soon!

Was the test successful?

Hello, apologies as I put my update on the other issue: https://github.com/open-policy-agent/gatekeeper/issues/1061

Hi @ritazh - It seems my suspicion was not correct, and removing the control-plane label did not help.

It’s really interesting that this is only affecting Gatekeeper, as we do have other tools with MWH and VWH which do not see this problem, and the traffic causing the errors is 100% coming from the konnectivity-agent pods in kube-system

I also took a look in konnectivity configmap and deployment manifest in one of our clusters to see if I could find a log format option, but I’m afraid I couldn’t find any. My main concern is that these are not coming in json format, so it causes a lot of spam for our fluentd instance to try to parse non-json log outputs as json.

We’re testing today and I will report back soon!

Was the test successful?

Hi @ritazh I believe that is incorrect. These errors also come on Kubernetes 1.22 for us, and also others have noted in this issue that they happen on K8s 1.21.

This is not just related to on Kubernetes 1.23 and 1.24 this is happening on all kuberenetes ( AWS EKS ) version 1.21

comment

Furthermore, https://github.com/kubernetes/kubernetes/issues/109022 clearly indicates the errors coming from 127.0.0.1. The original post of this issue does not indicate 127.0.0.1, but rather has the IP addresses masked as x.x.x.x which leads me to believe that the OP is experiencing this from their 10.x.x.x/8 subnet, the same as myself.

@ritazh Here is the error log … redacted some information for security purpose.

gatekeeper version is 3.8.1

  k logs -n gatekeeper deploy/gatekeeper-controller-manager -f
  Found 3 pods, using pod/gatekeeper-controller-manager-xxxxxxx-ldsc7
2022/09/08 01:14:32 http: TLS handshake error from x.x.x.x:49070: EOF
2022/09/08 01:46:37 http: TLS handshake error from x.x.x.x:35184: EOF
2022/09/08 02:47:46 http: TLS handshake error from x.x.x.x:39938: EOF
2022/09/08 06:47:20 http: TLS handshake error from x.x.x.x:38652: EOF
2022/09/08 12:37:59 http: TLS handshake error from x.x.x.x:49956: EOF
2022/09/08 13:16:45 http: TLS handshake error from x.x.x.x:56032: EOF
2022/09/08 13:41:48 http: TLS handshake error from x.x.x.x:56232: EOF
2022/09/08 16:38:13 http: TLS handshake error from x.x.x.x:60828: EOF
2022/09/08 19:02:34 http: TLS handshake error from x.x.x.x:36744: EOF

sorry about the delayed response.

This is not just related to on Kubernetes 1.23 and 1.24 this is happening on all kuberenetes ( AWS EKS ) version 1.21

xref: https://github.com/open-policy-agent/gatekeeper/issues/866#issuecomment-1139356714