etcd: Struggle with etcd timeout, help

We use k8s in production, about 300 nodes. Serval days ago, the etcd cluster became abnormal, no leader can be elected and all the client requests timeout. Here is the node config:

/data/bin/etcd-3.2
    --name betcd05
    --initial-cluster betcd05=http://tzk05:3380,betcd03=http://tzk03:3380,betcd01=http://tzk01:3380,betcd04=http://tzk04:3380,betcd02=http://tzk02:3380
    --initial-advertise-peer-urls http://10.3.8.18:3380
    --data-dir /data/data/etcd-bay4
    --wal-dir /data/log/etcd-bay4
    --listen-peer-urls http://10.3.8.18:3380
    --listen-client-urls http://10.3.8.18:3379,http://127.0.0.1:3379
    --advertise-client-urls http://10.3.8.18:3379
    --initial-cluster-state existing
    --quota-backend-bytes=8589934592
    --heartbeat-interval '1000'
    --election-timeout '10000'

etcd version 3.2.14 Here is the master node endpoint status:

ETCDCTL_API=3 etcdctl --endpoints=tzk01:3379 endpoint status
tzk01:3379, 6c31ba6970ba5e26, 3.2.14, 3.4 GB, true, 346, 123773332

The master node log shows that the query time too long…

2018-01-17 18:51:46.388490 W | etcdserver: apply entries took too long [31.038059099s for 1 entries]
2018-01-17 18:51:46.388505 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:51:59.666084 W | etcdserver: apply entries took too long [12.772745279s for 1 entries]
2018-01-17 18:51:59.666101 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:52:13.236024 W | etcdserver: apply entries took too long [13.016438597s for 1 entries]
2018-01-17 18:52:13.236046 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:52:26.511690 W | etcdserver: apply entries took too long [12.756021442s for 1 entries]
2018-01-17 18:52:26.511711 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:52:39.861618 W | etcdserver: apply entries took too long [12.822592077s for 1 entries]
2018-01-17 18:52:39.861636 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:52:54.475787 W | etcdserver: apply entries took too long [14.083658231s for 1 entries]
2018-01-17 18:52:54.475806 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:53:07.687890 W | etcdserver: apply entries took too long [12.71080088s for 1 entries]
2018-01-17 18:53:07.687908 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:53:20.919569 W | etcdserver: apply entries took too long [12.708351937s for 1 entries]
2018-01-17 18:53:20.919585 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:53:21.169769 W | etcdserver: apply entries took too long [103.492288ms for 1 entries]
2018-01-17 18:53:21.169795 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:53:40.588217 W | etcdserver: apply entries took too long [19.100170034s for 1 entries]
2018-01-17 18:53:40.588232 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:53:55.389231 W | etcdserver: apply entries took too long [14.270018598s for 1 entries]
2018-01-17 18:53:55.389248 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:53:55.408950 I | mvcc: store.index: compact 2476681628
2018-01-17 18:53:55.836416 W | etcdserver: apply entries took too long [427.854965ms for 1 entries]
2018-01-17 18:53:55.836435 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:54:02.478435 I | mvcc: finished scheduled compaction at 2476681628 (took 6.641971303s)
2018-01-17 18:54:09.624533 W | etcdserver: apply entries took too long [13.735092749s for 1 entries]
2018-01-17 18:54:09.624553 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:54:24.352842 W | etcdserver: apply entries took too long [14.197159572s for 1 entries]
2018-01-17 18:54:24.352859 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:54:38.258577 W | etcdserver: apply entries took too long [13.372680952s for 1 entries]
2018-01-17 18:54:38.258593 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:54:52.069577 W | etcdserver: apply entries took too long [13.249941442s for 1 entries]
2018-01-17 18:54:52.069595 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:55:06.132372 W | etcdserver: apply entries took too long [13.491814966s for 1 entries]
2018-01-17 18:55:06.132391 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:55:21.022463 I | wal: segmented wal file /data/log/etcd-bay4/0000000000000405-000000000760e9ba.wal is created
2018-01-17 18:55:25.017074 W | etcdserver: apply entries took too long [18.367109561s for 1 entries]
2018-01-17 18:55:25.017427 W | etcdserver: avoid queries with large range/delete range!
2018-01-17 18:55:38.776670 W | etcdserver: apply entries took too long [13.190163366s for 1 entries]
2018-01-17 18:55:38.776686 W | etcdserver: avoid queries with large range/delete range!

Here are the master node log and metrics files: master-node.log master-node-metrics.log

Disk, memory, network, all is OK. The problem was strange and made a disaster. If you need more info, please tell me.