etcd: rejected connection from "192.168.3.169:53244" (error "tls: first record does not look like a TLS handshake", ServerName "")

etcd: 3.3.8 OS:CentOS Linux release 7.5.1804 (Core) Arch: linux/amd64

etcd cluster:

etcd1: 192.168.3.162
etcd2: 192.168.3.168
etcd3: 192.168.3.169

Self-signed certificates：

{
    "CN": "kubernetes",
    "hosts": [
      "127.0.0.1",
            "192.168.3.162",
            "192.168.3.168",
            "192.168.3.169",
            "192.168.3.170",
            "192.168.3.171",
            "192.168.3.172",
            "192.168.3.173",
            "10.0.0.1",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "ChengDu",
            "L": "ChengDu",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

etcd1.service

[Unit]
Description=Etcd Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd --name etcd1 \
    --initial-advertise-peer-urls https://192.168.3.162:2380 \
    --listen-peer-urls https://192.168.3.162:2380 \
    --listen-client-urls https://192.168.3.162:2379,https://127.0.0.1:2379 \
    --advertise-client-urls https://192.168.3.162:2379 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-cluster etcd1=https://192.168.3.162:2380,etcd2=https://192.168.3.168:2380,etcd3=https://192.168.3.169:2380, \
    --initial-cluster-state new \
    --data-dir=/var/lib/etcd \
    --client-cert-auth \
    --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
    --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
    --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
    --peer-client-cert-auth \
    --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
    --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
    --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem

[Install]
WantedBy=multi-user.target

etcd2.service

[Unit]
Description=Etcd Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd --name etcd2 \
    --initial-advertise-peer-urls https://192.168.3.168:2380 \
    --listen-peer-urls https://192.168.3.168:2380 \
    --listen-client-urls https://192.168.3.168:2379,https://127.0.0.1:2379 \
    --advertise-client-urls https://192.168.3.168:2379 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-cluster etcd1=https://192.168.3.162:2380,etcd2=https://192.168.3.168:2380,etcd3=https://192.168.3.169:2380, \
    --initial-cluster-state new \
    --data-dir=/var/lib/etcd \
    --client-cert-auth \
    --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
    --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
    --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
    --peer-client-cert-auth \
    --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
    --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
    --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem

[Install]
WantedBy=multi-user.target

etcd3.service

[Unit]
Description=Etcd Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/bin/etcd --name etcd3 \
    --initial-advertise-peer-urls https://192.168.3.169:2380 \
    --listen-peer-urls https://192.168.3.169:2380 \
    --listen-client-urls https://192.168.3.169:2379,https://127.0.0.1:2379 \
    --advertise-client-urls https://192.168.3.169:2379 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-cluster etcd1=https://192.168.3.162:2380,etcd2=https://192.168.3.168:2380,etcd3=https://192.168.3.169:2380, \
    --initial-cluster-state new \
    --data-dir=/var/lib/etcd \
    --client-cert-auth \
    --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
    --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
    --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
    --peer-client-cert-auth \
    --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
    --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
    --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem

[Install]
WantedBy=multi-user.target

When I run systemctl start etcd, I get this error(from journalctl -xe):

Jul 11 04:44:38 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35500" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:38 etcd1 etcd[23500]: publish error: etcdserver: request timed out
Jul 11 04:44:38 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46496" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:38 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46500" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:38 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35510" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:38 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35508" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:38 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46504" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:38 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46508" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:38 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35518" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:38 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35516" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:38 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46512" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:38 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46516" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35526" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35524" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46520" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46524" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35534" (error "tls: oversized record received with length 21536", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: bab1199d24d460ee is starting a new election at term 925
Jul 11 04:44:39 etcd1 etcd[23500]: bab1199d24d460ee became candidate at term 926
Jul 11 04:44:39 etcd1 etcd[23500]: bab1199d24d460ee received MsgVoteResp from bab1199d24d460ee at term 926
Jul 11 04:44:39 etcd1 etcd[23500]: bab1199d24d460ee [logterm: 510, index: 2621] sent MsgVote request to e3b08f0ab9f3fe8 at term 926
Jul 11 04:44:39 etcd1 etcd[23500]: bab1199d24d460ee [logterm: 510, index: 2621] sent MsgVote request to 341f2c77ffb24eef at term 926
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35536" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35538" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46528" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46532" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35546" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35544" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46536" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46540" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35552" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35554" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46544" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46548" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35562" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35560" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46552" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46556" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46560" (error "tls: oversized record received with length 21536", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35568" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35570" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46564" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46568" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35578" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35576" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46572" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46576" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35584" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35586" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46578" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46584" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35594" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35592" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46588" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46592" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35600" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35602" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46600" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:39 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46596" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35608" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35610" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46604" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46608" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35616" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35618" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46612" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46616" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35626" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35624" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46620" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46624" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: bab1199d24d460ee is starting a new election at term 926
Jul 11 04:44:40 etcd1 etcd[23500]: bab1199d24d460ee became candidate at term 927
Jul 11 04:44:40 etcd1 etcd[23500]: bab1199d24d460ee received MsgVoteResp from bab1199d24d460ee at term 927
Jul 11 04:44:40 etcd1 etcd[23500]: bab1199d24d460ee [logterm: 510, index: 2621] sent MsgVote request to e3b08f0ab9f3fe8 at term 927
Jul 11 04:44:40 etcd1 etcd[23500]: bab1199d24d460ee [logterm: 510, index: 2621] sent MsgVote request to 341f2c77ffb24eef at term 927
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35634" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35632" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46628" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.169:46632" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35640" (error "tls: first record does not look like a TLS handshake", ServerName "")
Jul 11 04:44:40 etcd1 etcd[23500]: rejected connection from "192.168.3.168:35642" (error "tls: first record does not look like a TLS handshake", ServerName "")

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 21 (9 by maintainers)

Most upvoted comments

@wenjiaswe Did you ever find out what component was trying to connect without TLS? We’re running etcd 3.3.11 on-prem (without k8s), with manually generated certs. The etcd cluster seems to be behaving correctly, we can interact with the cluster using etcdctl and it is passing performance tests. However, we are still seeing the "tls: first record does not look like a TLS handshake" error in the logs every 10 seconds (I believe the 10 seconds is due to the transports TLS handshake timeout).

All our peer URL’s are https; new members join with https; there is not a single http address in the logs and we are not seeing any certificate errors. The errors only appear after "ready to serve client requests" appears in the logs: as the client/peer listeners only serve requests after this point. I believe the error is caused by a gRPC call to one of these listener endpoints without using TLS, but I am not sure what is making this call.

Do you have any idea what the root cause could be, and if it’s critical? Thanks!

tompntn on Jun 9, 2019

for anyone who comes across of the same issue in the future: check if you have both SSL & plain-text SRV records. Removing the latter fixed the issue and the subject logs disappeared.

sfuerte on Jun 5, 2022