kubernetes: Incorrect etcd version bundled with etcd docker image - k8s.gcr.io/etcd:3.5.0-0

What happened?

I am running a Kubernetes v1.22.2 cluster on ppc64le which uses k8s.gcr.io/etcd:3.5.0-0 for etcd. The actual etcd version bundled with this docker image is v3.4.13 which seems incorrect. Because of this version mismatch, etcd cannot recognize the flags (–experimental-enable-distributed-tracing) required to enable OpenTelemetry as tracing was enabled in v3.5 onwards

# kubectl exec -it $(kubectl get pods -n kube-system| grep etcd |awk '{print $1}') -n kube-system -- etcd --version
etcd Version: 3.4.13
Git SHA: ae9734ed2
Go Version: go1.13.5
Go OS/Arch: linux/ppc64le

What did you expect to happen?

etcd should have worked with the Tracing flags but it does not recognize them so it fails causing the cluster to go down.

How can we reproduce it (as minimally and precisely as possible)?

NOTE - This is for ppc64le hardware Create a single node kubernetes cluster (v1.22.2), edit /etc/kubernetes/manifests/etcd.yaml and add the below flags -

--experimental-enable-distributed-tracing=true
--experimental-distributed-tracing-address=0.0.0.0:4317
--experimental-distributed-tracing-service-name=etcd

This will cause etcd to fail with the below error -

2021-10-28T11:34:21.460593118Z stderr F flag provided but not defined: -experimental-enable-distributed-tracing
2021-10-28T11:34:21.460620131Z stderr F Usage:
2021-10-28T11:34:21.460626933Z stderr F
2021-10-28T11:34:21.460632029Z stderr F   etcd [flags]
2021-10-28T11:34:21.460637777Z stderr F     Start an etcd server.

Anything else we need to know?

This error has been seen on a cluster created on ppc64le h/w

Kubernetes version

$ kubectl version
# kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/ppc64le"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:32:41Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/ppc64le"}

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"

$ uname -a
Linux piy-test-ot 4.18.0-305.19.1.el8_4.ppc64le #1 SMP Wed Sep 15 15:51:05 UTC 2021 ppc64le ppc64le ppc64le GNU/Linux

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)