kubeadm: "kubeadm-config" does not exist when calling "upgrade plan" with v1.27.8

What happened?

When trying to upgrade my cluster from version 1.26 to version 1.27 following the instructions on https://v1-27.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ I received a fatal error that the kubeadm-config ConfigMap does not exist while executing ‘kubeadm upgrade plan’.

root@k8s-esx-master:/etc/kubernetes# kubeadm upgrade plan v1.27.8
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade/config] In order to upgrade, a ConfigMap called "kubeadm-config" in the kube-system namespace must exist.
[upgrade/config] Without this information, 'kubeadm upgrade' won't know how to configure your upgraded cluster.

[upgrade/config] Next steps:
	- OPTION 1: Run 'kubeadm config upload from-flags' and specify the same CLI arguments you passed to 'kubeadm init' when you created your control-plane.
	- OPTION 2: Run 'kubeadm config upload from-file' and specify the same config file you passed to 'kubeadm init' when you created your control-plane.
	- OPTION 3: Pass a config file to 'kubeadm upgrade' using the --config flag.

[upgrade/config] FATAL: the ConfigMap "kubeadm-config" in the kube-system namespace used for getting configuration information was not found
To see the stack trace of this error execute with --v=5 or higher

Looking at the ConfigMap I can see it’s there and looking fine:

root@k8s-esx-master:/etc/kubernetes/pki# kubectl -n kube-system get cm kubeadm-config -o yaml
apiVersion: v1
data:
  ClusterConfiguration: |
    apiServer:
      extraArgs:
        authorization-mode: Node,RBAC
      timeoutForControlPlane: 4m0s
    apiVersion: kubeadm.k8s.io/v1beta3
    certificatesDir: /etc/kubernetes/pki
    clusterName: kubernetes
    controllerManager: {}
    dns: {}
    etcd:
      local:
        dataDir: /var/lib/etcd
    imageRepository: registry.k8s.io
    kind: ClusterConfiguration
    kubernetesVersion: v1.26.4
    networking:
      dnsDomain: cluster.local
      serviceSubnet: 10.96.0.0/12
    scheduler: {}
  ClusterStatus: |
    apiEndpoints:
      k8s-esx-master:
        advertiseAddress: 192.168.0.180
        bindPort: 6443
    apiVersion: kubeadm.k8s.io/v1beta2
    kind: ClusterStatus
kind: ConfigMap
metadata:
  creationTimestamp: "2019-04-08T19:16:18Z"
  name: kubeadm-config
  namespace: kube-system
  resourceVersion: "501536465"
  uid: c916a385-5a32-11e9-8102-000c296bd767

I searched through several other issues but every advice I tried didn’t work.

Anybody has an idea how to resolve this issue?

Thanks in advance and rest regards

What did you expect to happen?

I would have expected to see the upgrade information.

How can we reproduce it (as minimally and precisely as possible)?

Try to upgrade a 1.26.4 cluster to 1.27.8 using kubeadm.

Anything else we need to know?

No response

Kubernetes version

root@k8s-esx-master:/etc/kubernetes/pki# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.4", GitCommit:"f89670c3aa4059d6999cb42e23ccb4f0b9a03979", GitTreeState:"clean", BuildDate:"2023-04-12T12:13:53Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.4", GitCommit:"f89670c3aa4059d6999cb42e23ccb4f0b9a03979", GitTreeState:"clean", BuildDate:"2023-04-12T12:05:35Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}

root@k8s-esx-master:/etc/kubernetes/pki# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.8", GitCommit:"66fee42707cd7f5a89f1987f7cb81b02dd19161c", GitTreeState:"clean", BuildDate:"2023-11-15T16:57:43Z", GoVersion:"go1.20.11", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"

$ uname -a
Linux k8s-esx-master 5.4.0-167-generic kubernetes/kubernetes#184-Ubuntu SMP Tue Oct 31 09:21:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)