kubernetes: kube-apiserver log always has TLS handshake error

I am also seeing this issue in my cluster. Can we please reopen it? /reopen

This can be solved by switch LB health check from TCP to SSL. Here are more details about this issue: https://github.com/kubernetes-incubator/kube-aws/issues/295 Try to add: “option ssl-hello-chk” as described in docs: https://www.haproxy.com/documentation/aloha/10-0/traffic-management/lb-layer7/health-checks/.

I also receive lot of tls: client offered an unsupported records. I discover these are generated by HAproxy healt check ssl-hello-chk witch use only SSLv3, while apiserver require TLS. I switch to check-ssl which use openssl implemention and disable the verification because the certificates are self signed, generated by kubeadm. Here are my changes:

backend apiserver
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    #option ssl-hello-chk
    balance     roundrobin
        server PZN-BU-ADM-01 10.x.x.11:6443 check check-ssl verify none
        server PZN-BU-ADM-02 10.x.x12:6443 check check-ssl verify none
        server PZN-BU-ADM-03 10.x.x.13:6443 check check-ssl verify none

Have the same issue. /priority important-soon

I wonder why the case is issue is closed, I have been jumping from one link to another and I cannot see the fix for the issue, I have 3 masters behind HAPROXY, and I am suffering exactly from the same problem, and below is the config for the api pod: spec: containers:

• command:
  • kube-apiserver
  • –authorization-mode=Node,RBAC
  • –advertise-address=10.245.10.10
  • –allow-privileged=true
  • –client-ca-file=/etc/kubernetes/pki/ca.crt
  • –enable-admission-plugins=NodeRestriction,PodNodeSelector
  • –enable-bootstrap-token-auth=true
  • –etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
  • –etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
  • –etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
  • –etcd-servers=https://10.245.10.2:2379,https://10.245.10.3:2379,https://10.245.10.3:2379
  • –insecure-port=0
  • –kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
  • –kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
  • –kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  • –proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
  • –proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
  • –requestheader-allowed-names=front-proxy-client
  • –requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
  • –requestheader-extra-headers-prefix=X-Remote-Extra-
  • –requestheader-group-headers=X-Remote-Group
  • –requestheader-username-headers=X-Remote-User
  • –secure-port=6443
  • –service-account-key-file=/etc/kubernetes/pki/sa.pub
  • –service-cluster-ip-range=10.96.0.0/12
  • –tls-cert-file=/etc/kubernetes/pki/apiserver.crt
  • –tls-private-key-file=/etc/kubernetes/pki/apiserver.key image: k8s.gcr.io/kube-apiserver:v1.13.4 And I am getting these error : http: TLS handshake error from 10.245.10.5:60478: tls: first record does not look like a TLS handshake Where is the IP 10.245.10.5 is the LB IP

The same is happening, if kube-apiserver is behind AWS load balancer, which does TCP health checks by default. If you configure it to use HTTPS, it still does not fully resolve the problem, as if one uses --anonymous-auth=false, then the health probes will be failing, as AWS has no way to configure authentication header for the health checks.

Maybe the log level of this message could be changed?

Was there ever any resolution to this? I see my logs flooded with these errors. I am not running HA Proxy. I have a 6 node cluster 3 control plane and 3 worker nodes. I see the TLS errors on the worker nodes. Health of the cluster seems to be ok. Using certs generated by kubernetes during kubeadm init and all certs are valid…

kubelet[2849943]: I0119 08:48:11.744715 2849943 log.go:181] http: TLS handshake error

well to clarify, by “fixed” we mean its going to trace level so it doesn’t flood right? but technically the issue is still happening it just wont fill up kube-api logs it seems.

Yeah, it won’t flood by default. But given that the reason for this log message heavily depends on the environment you run on, there is nothing to fix in kube-apiserver itself.

Happens to me as well and im not using HAproxy

Fresh installed with kubeadm with v1.18.6, having 2 control plane nodes

From APIserver pods:

[kube-apiserver-nuc02] I0807 23:19:15.894341       1 log.go:172] http: TLS handshake error from 172.16.137.160:50864: EOF 
[kube-apiserver-nuc01] I0807 23:19:15.921473       1 log.go:172] http: TLS handshake error from 192.168.5.135:24189: EOF

From etcd pods:

[etcd-nuc02] 2020-08-07 23:19:15.896327 I | embed: rejected connection from "172.16.137.160:40178" (error "EOF", ServerName "") 
[etcd-nuc01] 2020-08-07 23:19:15.923966 I | embed: rejected connection from "192.168.5.135:13800" (error "EOF", ServerName "") 
[etcd-nuc02] 2020-08-07 23:19:15.920069 I | embed: rejected connection from "172.16.137.160:40200" (error "tls: client didn't provide a certificate", ServerName "") 
[etcd-nuc01] 2020-08-07 23:19:15.934204 I | embed: rejected connection from "192.168.5.135:26652" (error "tls: client didn't provide a certificate", ServerName "")

I am on kubeadm version :“v1.17.4”

Getting the TLS handshake error, which is causing my kube-apiserver to restart.

Flag --insecure-port has been deprecated, This flag will be removed in a future version. I0430 09:13:29.046060 1 server.go:596] external host was not specified, using 192.168.0.109 I0430 09:13:29.046715 1 server.go:150] Version: v1.17.4 I0430 09:13:29.682184 1 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,MutatingAdmissionWebhook,RuntimeClass. I0430 09:13:29.682326 1 plugins.go:161] Loaded 7 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,RuntimeClass,ResourceQuota. I0430 09:13:29.684790 1 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,MutatingAdmissionWebhook,RuntimeClass. I0430 09:13:29.685087 1 plugins.go:161] Loaded 7 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,RuntimeClass,ResourceQuota. I0430 09:13:29.689415 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:29.689634 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:29.713493 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:29.713534 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:29.735309 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:29.735559 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:29.831465 1 master.go:267] Using reconciler: lease I0430 09:13:29.832422 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:29.832629 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:29.853007 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:29.853214 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:29.870593 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:29.870879 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:29.889623 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:29.889919 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:29.912727 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:29.916148 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:29.943443 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:29.943510 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:29.963027 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:29.963287 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:29.990784 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:29.991049 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.011970 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.012195 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.043015 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.043580 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.060042 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.060247 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.078773 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.078838 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.101854 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.102291 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.119095 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.119611 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.138909 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.139155 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.178625 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.179108 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.210588 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.210846 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.232103 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.232516 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.254297 1 rest.go:115] the default service ipfamily for this cluster is: IPv4 I0430 09:13:30.497764 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.497846 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.514043 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.514236 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.536677 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.537077 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.548509 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.548541 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.592759 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.593235 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.619654 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.620018 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.640614 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.640989 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.657900 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.658346 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.676889 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.677564 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.680278 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.680613 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.712588 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.713347 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.729198 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.729610 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.744314 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.744538 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.763012 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.763289 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.783781 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.786494 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.816324 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.816431 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.834033 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.834072 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.848592 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.848900 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.864391 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.865217 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.886566 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.886839 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.901444 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.901812 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.920897 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.921173 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.938260 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.938309 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.957502 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.957549 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.979526 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.980037 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:30.994640 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:30.994880 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.009476 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.009517 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.024340 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.024796 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.049074 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.049122 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.065782 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.066019 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.080212 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.080242 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.092721 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.093324 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.106598 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.106795 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.125340 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.125530 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.169099 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.169220 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.185814 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.185856 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.199060 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.200899 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.247241 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.247662 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.265445 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.265494 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.285433 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.285614 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.309414 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.309541 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.330430 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.330697 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.342677 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.343239 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] W0430 09:13:31.664953 1 genericapiserver.go:409] Skipping API batch/v2alpha1 because it has no resources. W0430 09:13:31.686651 1 genericapiserver.go:409] Skipping API discovery.k8s.io/v1alpha1 because it has no resources. W0430 09:13:31.709837 1 genericapiserver.go:409] Skipping API node.k8s.io/v1alpha1 because it has no resources. W0430 09:13:31.765238 1 genericapiserver.go:409] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. W0430 09:13:31.774200 1 genericapiserver.go:409] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources. W0430 09:13:31.802621 1 genericapiserver.go:409] Skipping API storage.k8s.io/v1alpha1 because it has no resources. W0430 09:13:31.851526 1 genericapiserver.go:409] Skipping API apps/v1beta2 because it has no resources. W0430 09:13:31.852214 1 genericapiserver.go:409] Skipping API apps/v1beta1 because it has no resources. I0430 09:13:31.872846 1 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,MutatingAdmissionWebhook,RuntimeClass. I0430 09:13:31.872945 1 plugins.go:161] Loaded 7 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,RuntimeClass,ResourceQuota. I0430 09:13:31.875566 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.875738 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:31.887561 1 client.go:361] parsed scheme: "endpoint" I0430 09:13:31.887792 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 <nil>}] I0430 09:13:38.182360 1 dynamic_cafile_content.go:166] Starting request-header::/etc/kubernetes/pki/front-proxy-ca.crt I0430 09:13:38.182592 1 dynamic_cafile_content.go:166] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt I0430 09:13:38.183630 1 dynamic_serving_content.go:129] Starting serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key I0430 09:13:38.184672 1 secure_serving.go:178] Serving securely on [::]:6443 I0430 09:13:38.184891 1 tlsconfig.go:219] Starting DynamicServingCertificateController I0430 09:13:38.187130 1 crd_finalizer.go:263] Starting CRDFinalizer I0430 09:13:38.188074 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller I0430 09:13:38.188180 1 shared_informer.go:197] Waiting for caches to sync for cluster_authentication_trust_controller I0430 09:13:38.188359 1 apiservice_controller.go:94] Starting APIServiceRegistrationController I0430 09:13:38.188504 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller I0430 09:13:38.188673 1 controller.go:81] Starting OpenAPI AggregationController I0430 09:13:38.189600 1 available_controller.go:386] Starting AvailableConditionController I0430 09:13:38.189730 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller I0430 09:13:38.189829 1 autoregister_controller.go:140] Starting autoregister controller I0430 09:13:38.189964 1 cache.go:32] Waiting for caches to sync for autoregister controller I0430 09:13:38.211990 1 log.go:172] http: TLS handshake error from 192.168.5.30:35814: EOF I0430 09:13:38.272928 1 log.go:172] http: TLS handshake error from 192.168.5.30:35728: EOF I0430 09:13:38.561863 1 log.go:172] http: TLS handshake error from 192.168.5.30:35730: EOF I0430 09:13:38.562589 1 log.go:172] http: TLS handshake error from 192.168.5.30:35816: EOF I0430 09:13:38.565045 1 log.go:172] http: TLS handshake error from 192.168.5.30:35732: EOF I0430 09:13:38.565215 1 log.go:172] http: TLS handshake error from 192.168.5.30:35818: EOF I0430 09:13:38.565398 1 log.go:172] http: TLS handshake error from 192.168.5.30:35820: EOF I0430 09:13:38.565579 1 log.go:172] http: TLS handshake error from 192.168.5.30:35734: EOF I0430 09:13:38.569222 1 log.go:172] http: TLS handshake error from 192.168.5.30:35822: EOF I0430 09:13:38.570204 1 log.go:172] http: TLS handshake error from 192.168.5.30:35736: EOF I0430 09:13:38.570429 1 log.go:172] http: TLS handshake error from 192.168.5.30:35824: EOF I0430 09:13:38.571230 1 log.go:172] http: TLS handshake error from 192.168.5.30:35738: EOF I0430 09:13:38.572609 1 log.go:172] http: TLS handshake error from 192.168.5.30:35826: EOF I0430 09:13:38.573045 1 log.go:172] http: TLS handshake error from 192.168.5.30:35740: EOF I0430 09:13:38.573422 1 log.go:172] http: TLS handshake error from 192.168.5.30:35828: EOF I0430 09:13:38.573852 1 log.go:172] http: TLS handshake error from 192.168.5.30:35742: EOF I0430 09:13:38.574134 1 log.go:172] http: TLS handshake error from 192.168.5.30:35830: EOF I0430 09:13:38.574656 1 log.go:172] http: TLS handshake error from 192.168.5.30:35744: EOF I0430 09:13:38.574870 1 log.go:172] http: TLS handshake error from 192.168.5.30:35832: EOF I0430 09:13:38.574978 1 log.go:172] http: TLS handshake error from 192.168.5.30:35834: EOF I0430 09:13:38.575059 1 log.go:172] http: TLS handshake error from 192.168.5.30:35746: EOF I0430 09:13:38.575150 1 log.go:172] http: TLS handshake error from 192.168.5.30:35748: EOF I0430 09:13:38.575222 1 log.go:172] http: TLS handshake error from 192.168.5.30:35836: EOF I0430 09:13:38.575290 1 log.go:172] http: TLS handshake error from 192.168.5.30:35838: EOF I0430 09:13:38.575354 1 log.go:172] http: TLS handshake error from 192.168.5.30:35750: EOF I0430 09:13:38.575967 1 log.go:172] http: TLS handshake error from 192.168.5.30:35840: EOF I0430 09:13:38.576260 1 log.go:172] http: TLS handshake error from 192.168.5.30:35752: EOF I0430 09:13:38.576412 1 log.go:172] http: TLS handshake error from 192.168.5.30:35842: EOF I0430 09:13:38.576858 1 log.go:172] http: TLS handshake error from 192.168.5.30:35844: EOF I0430 09:13:38.577099 1 log.go:172] http: TLS handshake error from 192.168.5.30:35754: EOF I0430 09:13:38.577197 1 log.go:172] http: TLS handshake error from 192.168.5.30:35846: EOF I0430 09:13:38.577339 1 log.go:172] http: TLS handshake error from 192.168.5.30:35756: EOF I0430 09:13:38.577537 1 log.go:172] http: TLS handshake error from 192.168.5.30:35848: EOF I0430 09:13:38.577680 1 log.go:172] http: TLS handshake error from 192.168.5.30:35850: EOF I0430 09:13:38.577810 1 log.go:172] http: TLS handshake error from 192.168.5.30:35758: EOF I0430 09:13:38.577939 1 log.go:172] http: TLS handshake error from 192.168.5.30:35852: EOF I0430 09:13:38.578145 1 log.go:172] http: TLS handshake error from 192.168.5.30:35760: EOF I0430 09:13:38.578238 1 log.go:172] http: TLS handshake error from 192.168.5.30:35854: EOF I0430 09:13:38.578498 1 log.go:172] http: TLS handshake error from 192.168.5.30:35762: EOF I0430 09:13:38.578701 1 log.go:172] http: TLS handshake error from 192.168.5.30:35856: EOF I0430 09:13:38.578892 1 log.go:172] http: TLS handshake error from 192.168.5.30:35764: EOF I0430 09:13:38.579075 1 log.go:172] http: TLS handshake error from 192.168.5.30:35858: EOF I0430 09:13:38.579803 1 log.go:172] http: TLS handshake error from 192.168.5.30:35766: EOF I0430 09:13:38.580198 1 log.go:172] http: TLS handshake error from 192.168.5.30:35860: EOF I0430 09:13:38.580597 1 log.go:172] http: TLS handshake error from 192.168.5.30:35862: EOF I0430 09:13:38.585606 1 log.go:172] http: TLS handshake error from 192.168.5.30:35768: EOF E0430 09:13:38.553859 1 controller.go:151] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.0.109, ResourceVersion: 0, AdditionalErrorMsg: I0430 09:13:38.253981 1 customresource_discovery_controller.go:208] Starting DiscoveryController I0430 09:13:38.253996 1 naming_controller.go:288] Starting NamingConditionController I0430 09:13:38.254010 1 establishing_controller.go:73] Starting EstablishingController I0430 09:13:38.254026 1 nonstructuralschema_controller.go:191] Starting NonStructuralSchemaConditionController I0430 09:13:38.254047 1 apiapproval_controller.go:185] Starting KubernetesAPIApprovalPolicyConformantConditionController I0430 09:13:38.254115 1 dynamic_cafile_content.go:166] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt I0430 09:13:38.254132 1 dynamic_cafile_content.go:166] Starting request-header::/etc/kubernetes/pki/front-proxy-ca.crt I0430 09:13:38.255898 1 crdregistration_controller.go:111] Starting crd-autoregister controller I0430 09:13:38.627246 1 log.go:172] http: TLS handshake error from 192.168.5.30:35864: EOF I0430 09:13:38.629470 1 shared_informer.go:197] Waiting for caches to sync for crd-autoregister I0430 09:13:38.629761 1 log.go:172] http: TLS handshake error from 192.168.5.30:35770: EOF I0430 09:13:38.630508 1 log.go:172] http: TLS handshake error from 192.168.5.30:35866: EOF I0430 09:13:38.630841 1 log.go:172] http: TLS handshake error from 192.168.5.30:35772: EOF I0430 09:13:38.632333 1 log.go:172] http: TLS handshake error from 192.168.5.30:35868: EOF I0430 09:13:38.635426 1 log.go:172] http: TLS handshake error from 192.168.5.30:35774: EOF I0430 09:13:38.636151 1 log.go:172] http: TLS handshake error from 192.168.5.30:35870: EOF I0430 09:13:38.636830 1 log.go:172] http: TLS handshake error from 192.168.5.30:35872: EOF I0430 09:13:38.637462 1 log.go:172] http: TLS handshake error from 192.168.5.30:35776: EOF I0430 09:13:38.639845 1 log.go:172] http: TLS handshake error from 192.168.5.30:35874: EOF I0430 09:13:38.640215 1 log.go:172] http: TLS handshake error from 192.168.5.30:35778: EOF I0430 09:13:38.640320 1 log.go:172] http: TLS handshake error from 192.168.5.30:35876: EOF I0430 09:13:38.640512 1 log.go:172] http: TLS handshake error from 192.168.5.30:35780: EOF I0430 09:13:38.685948 1 log.go:172] http: TLS handshake error from 192.168.5.30:35878: EOF I0430 09:13:38.686108 1 log.go:172] http: TLS handshake error from 192.168.5.30:35880: EOF I0430 09:13:38.686197 1 log.go:172] http: TLS handshake error from 192.168.5.30:35782: EOF I0430 09:13:38.686346 1 log.go:172] http: TLS handshake error from 192.168.5.30:35882: EOF I0430 09:13:38.686969 1 log.go:172] http: TLS handshake error from 192.168.5.30:35784: EOF I0430 09:13:38.687069 1 log.go:172] http: TLS handshake error from 192.168.5.30:35884: EOF I0430 09:13:38.687223 1 log.go:172] http: TLS handshake error from 192.168.5.30:35786: EOF I0430 09:13:38.687304 1 log.go:172] http: TLS handshake error from 192.168.5.30:35886: EOF I0430 09:13:38.687372 1 log.go:172] http: TLS handshake error from 192.168.5.30:35788: EOF **I0430 09:13:38.687629 1 log.go:172] http: TLS handshake error from 192.168.5.30:35888: EOF I0430 09:13:38.687798 1 log.go:172] http: TLS handshake error from 192.168.5.30:35890: EOF I0430 09:13:38.687896 1 log.go:172] http: TLS handshake error from 192.168.5.30:35790: EOF** I0430 09:13:38.687997 1 log.go:172] http: TLS handshake error from 192.168.5.30:35792: EOF I0430 09:13:38.688091 1 log.go:172] http: TLS handshake error from 192.168.5.30:35892: EOF I0430 09:13:38.688177 1 log.go:172] http: TLS handshake error from 192.168.5.30:35794: EOF I0430 09:13:38.688261 1 log.go:172] http: TLS handshake error from 192.168.5.30:35894: EOF I0430 09:13:38.688349 1 log.go:172] http: TLS handshake error from 192.168.5.30:35796: EOF I0430 09:13:38.688434 1 log.go:172] http: TLS handshake error from 192.168.5.30:35896: EOF I0430 09:13:38.688522 1 log.go:172] http: TLS handshake error from 192.168.5.30:35798: EOF I0430 09:13:38.688642 1 log.go:172] http: TLS handshake error from 192.168.5.30:35898: EOF I0430 09:13:38.688876 1 log.go:172] http: TLS handshake error from 192.168.5.30:35800: EOF I0430 09:13:38.689216 1 log.go:172] http: TLS handshake error from 192.168.5.30:35802: EOF I0430 09:13:38.689324 1 log.go:172] http: TLS handshake error from 192.168.5.30:35804: EOF I0430 09:13:38.689440 1 log.go:172] http: TLS handshake error from 192.168.5.30:35900: EOF I0430 09:13:38.689548 1 log.go:172] http: TLS handshake error from 192.168.5.30:35806: EOF I0430 09:13:38.689718 1 log.go:172] http: TLS handshake error from 192.168.5.30:35902: EOF I0430 09:13:38.689837 1 log.go:172] http: TLS handshake error from 192.168.5.30:35904: EOF I0430 09:13:38.689930 1 log.go:172] http: TLS handshake error from 192.168.5.30:35808: EOF I0430 09:13:38.690019 1 log.go:172] http: TLS handshake error from 192.168.5.30:35906: EOF I0430 09:13:38.690100 1 log.go:172] http: TLS handshake error from 192.168.5.30:35810: EOF I0430 09:13:38.690318 1 log.go:172] http: TLS handshake error from 192.168.5.30:35812: EOF I0430 09:13:38.690467 1 log.go:172] http: TLS handshake error from 192.168.5.30:35724: EOF I0430 09:13:38.690864 1 log.go:172] http: TLS handshake error from 192.168.5.30:35726: EOF I0430 09:13:38.253961 1 controller.go:85] Starting OpenAPI controller I0430 09:13:38.734400 1 shared_informer.go:204] Caches are synced for crd-autoregister I0430 09:13:38.762650 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io I0430 09:13:38.834662 1 shared_informer.go:204] Caches are synced for cluster_authentication_trust_controller I0430 09:13:38.834755 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller I0430 09:13:38.858416 1 cache.go:39] Caches are synced for AvailableConditionController controller I0430 09:13:38.859250 1 cache.go:39] Caches are synced for autoregister controller E0430 09:13:38.921389 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"} E0430 09:13:38.921765 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"} I0430 09:13:39.182402 1 controller.go:107] OpenAPI AggregationController: Processing item I0430 09:13:39.182431 1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue). I0430 09:13:39.182443 1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue). I0430 09:13:39.190561 1 storage_scheduling.go:142] all system priority classes are created successfully or already exist. I0430 09:14:56.859520 1 dynamic_cafile_content.go:181] Shutting down request-header::/etc/kubernetes/pki/front-proxy-ca.crt I0430 09:14:56.859575 1 apiapproval_controller.go:197] Shutting down KubernetesAPIApprovalPolicyConformantConditionController I0430 09:14:56.859584 1 establishing_controller.go:84] Shutting down EstablishingController I0430 09:14:56.859599 1 naming_controller.go:299] Shutting down NamingConditionController I0430 09:14:56.859610 1 customresource_discovery_controller.go:219] Shutting down DiscoveryController I0430 09:14:56.859637 1 dynamic_cafile_content.go:181] Shutting down request-header::/etc/kubernetes/pki/front-proxy-ca.crt I0430 09:14:56.859649 1 dynamic_cafile_content.go:181] Shutting down client-ca-bundle::/etc/kubernetes/pki/ca.crt I0430 09:14:56.859683 1 controller.go:122] Shutting down OpenAPI controller I0430 09:14:56.859698 1 cluster_authentication_trust_controller.go:463] Shutting down cluster_authentication_trust_controller controller I0430 09:14:56.859711 1 autoregister_controller.go:164] Shutting down autoregister controller I0430 09:14:56.859722 1 available_controller.go:398] Shutting down AvailableConditionController I0430 09:14:56.859738 1 apiservice_controller.go:106] Shutting down APIServiceRegistrationController I0430 09:14:56.859748 1 crd_finalizer.go:275] Shutting down CRDFinalizer I0430 09:14:56.859804 1 controller.go:87] Shutting down OpenAPI AggregationController I0430 09:14:56.860328 1 crdregistration_controller.go:142] Shutting down crd-autoregister controller I0430 09:14:56.859579 1 nonstructuralschema_controller.go:203] Shutting down NonStructuralSchemaConditionController I0430 09:14:56.860418 1 tlsconfig.go:234] Shutting down DynamicServingCertificateController I0430 09:14:56.860440 1 dynamic_serving_content.go:144] Shutting down serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key I0430 09:14:56.860452 1 dynamic_cafile_content.go:181] Shutting down client-ca-bundle::/etc/kubernetes/pki/ca.crt I0430 09:14:56.859523 1 controller.go:180] Shutting down kubernetes service endpoint reconciler I0430 09:14:56.865291 1 secure_serving.go:222] Stopped listening on [::]:6443 E0430 09:14:56.883346 1 controller.go:183] Get https://localhost:6443/api/v1/namespaces/default/endpoints/kubernetes: dial tcp 127.0.0.1:6443: connect: connection refused Above is the comeplete log of kube-apiserver

Here 192.168.5.30 is the api LB ip address of HAproxy.

The HAproxy config:

`frontend kubernetes bind 192.168.5.30:8443 option tcplog mode tcp default_backend kubernetes-master-nodes

backend kubernetes-master-nodes mode tcp balance roundrobin option tcp-check server master-1 192.168.5.11:6443 check fall 3 rise 2 server master-2 192.168.5.12:6443 check fall 3 rise 2 `

Can someone pls guide me to the solution to fix this issue.