etcd: It's possible to get "wal: max entry size limit exceeded" with recommended values

What happened?

I run clean etcd node as followed: etcd --max-request-bytes=10485760

Then through the go client I put to key k1 value consists of 10*1024*1024-27 bytes.

Then I stopped server and try to start it again, but it failed with error wal: max entry size limit exceeded (https://github.com/etcd-io/etcd/blob/main/server/storage/wal/decoder.go#L88).

What did you expect to happen?

1. It should not be possible to write more data than etcd can read then.
2. It should be obvious how much data (key + value) in bytes I’m able to write. I set --max-request-bytes to 10mb, but in fact was able to pass 25bytes less data (2bytes to key, 10mb-27bytes for data). Maybe internal overhead should not be part of validation?
3. It should not be possible to set --max-request-bytes to value so high that etcd will allow to write more than it can read then. OR the WAL limit should be configurable.

How can we reproduce it (as minimally and precisely as possible)?

Run clean etcd instance: etcd --max-request-bytes=10485760

Run this go-code:

package main

import (
	"context"
	"go.etcd.io/etcd/clientv3"
)

func main() {
	length := 10*1024*1024 - 27
	b := make([]byte, length)
	for i := 0; i < length; i++ {
		b[i] = 'a'
	}

	cli, err := clientv3.New(clientv3.Config{
		Endpoints:          []string{"http://127.0.0.1:2379"},
		MaxCallSendMsgSize: length + 1024,
	})

	if err != nil {
		panic(err)
	}

	_, err = cli.Put(context.Background(), "k1", string(b))
	if err != nil {
		panic(err)
	}
}

Anything else we need to know?

Let’s assume you run code above, you will not be able to restart server. But you can call etcd snap save while server is still running, delete all WAL files and the start server and receive saved value.

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.5.4
Git SHA: 08407ff76
Go Version: go1.18.1
Go OS/Arch: darwin/amd64

$ etcdctl version
etcdctl version: 3.5.4
API version: 3.5

Etcd configuration (command line flags or environment variables)

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
+------------------+---------+---------+-----------------------+-----------------------+------------+
|        ID        | STATUS  |  NAME   |      PEER ADDRS       |     CLIENT ADDRS      | IS LEARNER |
+------------------+---------+---------+-----------------------+-----------------------+------------+
| 8e9e05c52164694d | started | default | http://localhost:2380 | http://localhost:2379 |      false |
+------------------+---------+---------+-----------------------+-----------------------+------------+

$ etcdctl --endpoints=<member list> endpoint status -w table
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|       ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://localhost:2379 | 8e9e05c52164694d |   3.5.4 |   25 kB |      true |      false |         2 |          4 |                  4 |        |
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Relevant log output

{"level":"fatal","ts":"2022-05-09T13:19:04.630+0300","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"wal: max entry size limit exceeded","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\t/private/tmp/etcd-20220424-57243-1ka6pvw/server/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/private/tmp/etcd-20220424-57243-1ka6pvw/server/etcdmain/main.go:40\nmain.main\n\t/private/tmp/etcd-20220424-57243-1ka6pvw/server/main.go:32\nruntime.main\n\t/usr/local/Cellar/go/1.18.1/libexec/src/runtime/proc.go:250"}