etcd: Rejected connection in ETCD logs from localhost every 10 seconds

I’ve noticed that this happens when you hit the “/healthz” endpoint on the kubernetes API Server for some reason. It doesn’t seem to be an issue but when you’re trying to debug something it wastes a few hours of your time ha

Been doing a little more debugging, it appears to be this call here:

• https://github.com/kubernetes/kubernetes/blob/v1.8.7/cmd/kube-apiserver/app/server.go#L278
• https://github.com/kubernetes/apiserver/blob/kubernetes-1.8.7/pkg/storage/etcd3/preflight/checks.go#L44

My kops master nodes start kube-apiserver up with the flag --etcd-servers=https://127.0.0.1:4001. Trawling through the code, the request ends up as my below example:

import (
	"fmt"
	"net"
	"time"
)

func main() {
	conn, err := net.DialTimeout("tcp", "127.0.0.1:4001", 1*time.Second)
	if err != nil {
		fmt.Printf("failed, %v", err)
	} else {
		conn.Close()
		fmt.Printf("successful\n") // I return successfully here, no errors.
	}
}

I can execute that to replicate the behaviour and logs in etcd exactly, where the above code gets a successful response however etcd reports the rejected connection.

Probably there are other client requests coming with wrong certs, thus being closed. Still the error message is not descriptive, which seems addressed in https://go-review.googlesource.com/c/go/+/86715/7/src/crypto/tls/tls.go.

Thank you for this issue. I’ve been tolling for two days to understand why kubectl exec fails on my 1.10 deployment (works on 11, 12, 13…). Kept seeing the EOF errors regarding health checks. Really bad when an AWS ALB is fronting your cluster, hitting the health checks every few seconds. If this is a false positive, it’s a doozy! Put me on a deep-dive this morning that sounds like I can ignore. Back to the other issue 😃

Hey, apologies for the late update!

I’ve been doing some debugging and found the following:

• It appears that the calls are originating from kubelet. When I stop the kubelet service and leave all other services operating, the rejected connection logs in etcd immediately stop.
• When running Kubernetes(/kubelet) v1.8.7 and ETCD v3.2.15, I do not see the logs.
• When running Kubernetes(/kubelet) v1.8.7 and ETCD v3.3.0, I see the logs.
• I bumped up the verbosity for kubelet and noticed various HTTP Probe logs, so tested them out:

kubelet[26610]: I0209 14:42:52.556308   26610 prober.go:160] HTTP-Probe Host: http://127.0.0.1, Port: 8080, Path: /healthz
kubelet[26610]: I0209 14:42:52.556338   26610 prober.go:163] HTTP-Probe Headers: map[]
kubelet[26610]: I0209 14:42:52.557460   26610 http.go:96] Probe succeeded for http://127.0.0.1:8080/healthz, Response: {200 OK 200 HTTP/1.1 1 1 map[Date:[Fri, 09 Feb 2018 14:42:52 GMT] Content-Length:[2] Content-Type:[text/plain; charset=utf-8]] 0xc4206a8da0 2 [] true false map[] 0xc42197dc00 <nil>}
kubelet[26610]: I0209 14:42:52.557499   26610 prober.go:113] Liveness probe for "kube-apiserver-ip-10-100-0-134.eu-west-2.compute.internal_kube-system(1ab242f3aecc4235bc25d68db8832ba5):kube-apiserver" succeeded

With both ETCD v3.2.15 and ETCD v3.3.0 Clusters I ran curl http://127.0.0.1:8080/healthz on a master node. On both, I get a HTTP 200 OK response and the response content ok is returned. By making this curl request, ETCD v3.3.0 returns the logs:

2018-02-09 14:42:52.557170 I | embed: rejected connection from "127.0.0.1:38132" (error "EOF", ServerName "")

So it’s caused when kubelet is performing liveness probes against different services and hits the kube-apiserver unauthenticated port. I’m now lost from here 😛