kubeadm: kubeadm upgrade from v1.28.0 to v1.28.3 fails

@liggitt Am I reading it wrong? shouldn’t this codec generate defaults?

It applies defaults it knows about, which are defaulting functions registered into the codec. Whether defaulting functions are registered or not depends on which packages are linked into the binary. The defaulting functions for core APIs are defined in k8s.io/kubernetes/… API packages and only intended for use by kube-apiserver

kubeadm version: &version.Info{Major:“1”, Minor:“28”, GitVersion:“v1.28.0”, GitCommit:“855e7c48de7388eb330da0f8d9d2394ee818fb8d”, GitTreeState:“clean”, BuildDate:“2023-08-15T10:20:15Z”, GoVersion:“go1.20.7”, Compiler:“gc”, Platform:“linux/amd64”}

we have exactly the same binary, but i’m getting a different etcd.yaml (minus the IP diff). mine does not have the problematic defaults like successThreshold: 1, dnsPolicy: ClusterFirst.

we saw similar strange behavior when the bug was found.

@@ -2,7 +2,7 @@ apiVersion: v1
 kind: Pod
 metadata:
   annotations:
-    kubeadm.kubernetes.io/etcd.advertise-client-urls: https://10.0.2.15:2379
+    kubeadm.kubernetes.io/etcd.advertise-client-urls: https://192.168.56.106:2379
   creationTimestamp: null
   labels:
     component: etcd
@@ -13,19 +13,19 @@ spec:
   containers:
   - command:
     - etcd
-    - --advertise-client-urls=https://10.0.2.15:2379
+    - --advertise-client-urls=https://192.168.56.106:2379
     - --cert-file=/etc/kubernetes/pki/etcd/server.crt
     - --client-cert-auth=true
     - --data-dir=/var/lib/etcd
     - --experimental-initial-corrupt-check=true
     - --experimental-watch-progress-notify-interval=5s
-    - --initial-advertise-peer-urls=https://10.0.2.15:2380
-    - --initial-cluster=lubo-it=https://10.0.2.15:2380
+    - --initial-advertise-peer-urls=https://192.168.56.106:2380
+    - --initial-cluster=ubuntu=https://192.168.56.106:2380
     - --key-file=/etc/kubernetes/pki/etcd/server.key
-    - --listen-client-urls=https://127.0.0.1:2379,https://10.0.2.15:2379
+    - --listen-client-urls=https://127.0.0.1:2379,https://192.168.56.106:2379
     - --listen-metrics-urls=http://127.0.0.1:2381
-    - --listen-peer-urls=https://10.0.2.15:2380
-    - --name=lubo-it
+    - --listen-peer-urls=https://192.168.56.106:2380
+    - --name=ubuntu
     - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
     - --peer-client-cert-auth=true
     - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
@@ -43,6 +43,7 @@ spec:
         scheme: HTTP
       initialDelaySeconds: 10
       periodSeconds: 10
+      successThreshold: 1
       timeoutSeconds: 15
     name: etcd
     resources:
@@ -58,18 +59,26 @@ spec:
         scheme: HTTP
       initialDelaySeconds: 10
       periodSeconds: 10
+      successThreshold: 1
       timeoutSeconds: 15
+    terminationMessagePath: /dev/termination-log
+    terminationMessagePolicy: File
     volumeMounts:
     - mountPath: /var/lib/etcd
       name: etcd-data
     - mountPath: /etc/kubernetes/pki/etcd
       name: etcd-certs
+  dnsPolicy: ClusterFirst
+  enableServiceLinks: true
   hostNetwork: true
   priority: 2000001000
   priorityClassName: system-node-critical
+  restartPolicy: Always
+  schedulerName: default-scheduler
   securityContext:
     seccompProfile:
       type: RuntimeDefault
+  terminationGracePeriodSeconds: 30
   volumes:
   - hostPath:
       path: /etc/kubernetes/pki/etcd

this means the problem might happen for some 1.28.0 users, but not for others… either way, the workarounds should be applied and there isn’t much we can do like @chendave said.

thanks for the details, let’s keep this tickets open until more users upgrade to 1.28.3. we might have to add an entry about it in: https://k8s-docs.netlify.app/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/

can you share your kubeadm cluster configuration? hide any IP / DNS names if needed: kubectl get cm -n kube-system kubeadm-config

also if you are passing --config to init, share that as well please.

Hi @neolit123 Here it is. kubeadm-config:

apiVersion: v1
data:
  ClusterConfiguration: |
    apiServer:
      certSANs:
      - ubuntu
      - 192.168.56.106
      extraArgs:
        audit-log-maxage: "30"
        audit-log-maxbackup: "10"
        audit-log-maxsize: "100"
        audit-log-path: /var/log/kubernetes/audit/audit.log
        audit-policy-file: /etc/kubernetes/audit-policy.yaml
        authorization-mode: Node,RBAC
        enable-admission-plugins: NodeRestriction
        profiling: "false"
      extraVolumes:
      - hostPath: /etc/kubernetes/audit-policy.yaml
        mountPath: /etc/kubernetes/audit-policy.yaml
        name: audit
        pathType: File
        readOnly: true
      - hostPath: /var/log/kubernetes/audit/
        mountPath: /var/log/kubernetes/audit/
        name: audit-log
        pathType: DirectoryOrCreate
      timeoutForControlPlane: 4m0s
    apiVersion: kubeadm.k8s.io/v1beta3
    certificatesDir: /etc/kubernetes/pki
    clusterName: kubernetes
    controlPlaneEndpoint: all.new.local:6443
    controllerManager:
      extraArgs:
        profiling: "false"
        terminated-pod-gc-threshold: "1000"
    dns:
      imageRepository: registry.k8s.io/coredns
    etcd:
      local:
        dataDir: /var/lib/etcd
    imageRepository: registry.k8s.io
    kind: ClusterConfiguration
    kubernetesVersion: v1.28.0
    networking:
      dnsDomain: cluster.local
      podSubnet: 10.128.0.0/14
      serviceSubnet: 172.30.0.0/16
    scheduler:
      extraArgs:
        profiling: "false"
kind: ConfigMap
metadata:
  creationTimestamp: "2023-11-08T13:26:45Z"
  name: kubeadm-config
  namespace: kube-system
  resourceVersion: "193"
  uid: 4ad93321-fad0-4021-97e9-d1501511d781

init-config:

apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
enableDebuggingHandlers: true
kind: KubeletConfiguration
podPidsLimit: 4096
protectKernelDefaults: true
readOnlyPort: 0
serializeImagePulls: false
---
apiServer:
  certSANs:
  - ubuntu
  - 192.168.56.106
  extraArgs:
    audit-log-maxage: '30'
    audit-log-maxbackup: '10'
    audit-log-maxsize: '100'
    audit-log-path: /var/log/kubernetes/audit/audit.log
    audit-policy-file: /etc/kubernetes/audit-policy.yaml
    enable-admission-plugins: NodeRestriction
    profiling: 'false'
  extraVolumes:
  - hostPath: /etc/kubernetes/audit-policy.yaml
    mountPath: /etc/kubernetes/audit-policy.yaml
    name: audit
    pathType: File
    readOnly: true
  - hostPath: /var/log/kubernetes/audit/
    mountPath: /var/log/kubernetes/audit/
    name: audit-log
    pathType: DirectoryOrCreate
    readOnly: false
apiVersion: kubeadm.k8s.io/v1beta3
controlPlaneEndpoint: all.new.local:6443
controllerManager:
  extraArgs:
    profiling: 'false'
    terminated-pod-gc-threshold: '1000'
dns:
  imageRepository: registry.k8s.io/coredns
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.28.0
networking:
  podSubnet: 10.128.0.0/14
  serviceSubnet: 172.30.0.0/16
scheduler:
  extraArgs:
    profiling: 'false'
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.56.106
nodeRegistration:
  criSocket: /var/run/containerd/containerd.sock
  kubeletExtraArgs:
    container-runtime-endpoint: unix:///run/containerd/containerd.sock
  taints: []
patches:
  directory: /etc/kubernetes/patches

IIRC, the bug will be triggered when the user’s cluster was already upgraded to v1.28.0-1 and then needs to be upgraded to v1.28.2+ or v1.29+.

For etcd, there is a new version v3.5.10. If the v1.28.x(next patch release) include that, it would be fixed in a tricky way if I understand correctly.

we patched it for 1.29 here: https://github.com/kubernetes/kubernetes/pull/120561

then we backported it for 1.28 here: https://github.com/kubernetes/kubernetes/pull/120605/commits/0c6a0c3f69bc20bc5422f554475fc260fb700e3b

that was on 14 of Sept and it should be part of 1.28.3, but not in 1.28.2 if i’m reading the history of the branch 1.28 correctly: https://github.com/kubernetes/kubernetes/commits/release-1.28?before=197e7579adb1bf180617bd3becc2aa4dcceb5291+35&branch=release-1.28&qualified_name=refs%2Fheads%2Frelease-1.28

so in theory there should be no problem for the .3 upgrade. but if .4 includes an actual etcd upgrade then there is no way for the hash issue to surface.

thanks for the logs, i will try to reproduce this locally. the workaround like you mentioned is to just skip the etcd upgrade. between 1.28.0 and 1.28.3 there is nothing to upgrade in etcd.

can you share full logs here or in a github Gist maybe? from what version are you upgrading from?

The issue might be fixed by the --etcd-upgrade flag.

if the flag is set to false, you mean?

The kubeadm v1.28.0 upgrades cluster successfully

we have our 1.27.latest -> 1.28.latest upgrade tests working fine: https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-upgrade-1-27-1-28

but etcd is not upgraded, because there is the same version between 1.27 and 1.28:

[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.28.3-22+197e7579adb1bf" (timeout: 5m0s)...
I1107 12:18:53.445360    4203 round_trippers.go:553] GET https://172.17.0.7:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-kinder-upgrade-control-plane-1?timeout=10s 200 OK in 4 milliseconds
I1107 12:18:53.450351    4203 round_trippers.go:553] GET https://172.17.0.7:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-kinder-upgrade-control-plane-1?timeout=10s 200 OK in 3 milliseconds
I1107 12:18:53.455022    4203 round_trippers.go:553] GET https://172.17.0.7:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-kinder-upgrade-control-plane-1?timeout=10s 200 OK in 3 milliseconds
I1107 12:18:53.455712    4203 etcd.go:214] retrieving etcd endpoints from "kubeadm.kubernetes.io/etcd.advertise-client-urls" annotation in etcd Pods
I1107 12:18:53.461659    4203 round_trippers.go:553] GET https://172.17.0.7:6443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Detcd%2Ctier%3Dcontrol-plane 200 OK in 5 milliseconds
I1107 12:18:53.463060    4203 etcd.go:150] etcd endpoints read from pods: https://172.17.0.2:2379,https://172.17.0.6:2379,https://172.17.0.3:2379
I1107 12:18:53.477995    4203 etcd.go:262] etcd endpoints read from etcd: https://172.17.0.2:2379,https://172.17.0.3:2379,https://172.17.0.6:2379
I1107 12:18:53.478146    4203 etcd.go:168] update etcd endpoints: https://172.17.0.2:2379,https://172.17.0.3:2379,https://172.17.0.6:2379
[upgrade/etcd] Upgrading to TLS for etcd
I1107 12:18:53.809153    4203 round_trippers.go:553] GET https://172.17.0.7:6443/api/v1/namespaces/kube-system/pods/etcd-kinder-upgrade-control-plane-1?timeout=10s 200 OK in 4 milliseconds
I1107 12:18:53.812731    4203 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests3206296204/etcd.yaml"
[upgrade/staticpods] Preparing for "etcd" upgrade
I1107 12:18:53.815034    4203 etcd.go:214] retrieving etcd endpoints from "kubeadm.kubernetes.io/etcd.advertise-client-urls" annotation in etcd Pods
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
I1107 12:18:53.821659    4203 round_trippers.go:553] GET https://172.17.0.7:6443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Detcd%2Ctier%3Dcontrol-plane 200 OK in 6 milliseconds
I1107 12:18:53.823184    4203 etcd.go:150] etcd endpoints read from pods: https://172.17.0.2:2379,https://172.17.0.6:2379,https://172.17.0.3:2379
I1107 12:18:53.838740    4203 etcd.go:262] etcd endpoints read from etcd: https://172.17.0.2:2379,https://172.17.0.3:2379,https://172.17.0.6:2379
I1107 12:18:53.838771    4203 etcd.go:168] update etcd endpoints: https://172.17.0.2:2379,https://172.17.0.3:2379,https://172.17.0.6:2379
I1107 12:18:53.838785    4203 etcd.go:588] [etcd] attempting to see if all cluster endpoints ([https://172.17.0.2:2379 https://172.17.0.3:2379 https://172.17.0.6:2379]) are available 1/10
[upgrade/etcd] Waiting for etcd to become available

our 1.28.latest -> 1.29.latest upgrade works as well, where actual etcd upgrade happens: https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-upgrade-1-28-latest

I1107 13:11:26.312401    3826 etcd.go:264] etcd endpoints read from etcd: https://172.17.0.5:2379,https://172.17.0.3:2379,https://172.17.0.6:2379
I1107 13:11:26.312428    3826 etcd.go:170] update etcd endpoints: https://172.17.0.5:2379,https://172.17.0.3:2379,https://172.17.0.6:2379
I1107 13:11:26.565664    3826 round_trippers.go:553] GET https://172.17.0.7:6443/api/v1/namespaces/kube-system/pods/etcd-kinder-upgrade-control-plane-1?timeout=10s 200 OK in 3 milliseconds
[upgrade/staticpods] Preparing for "etcd" upgrade
I1107 13:11:26.567768    3826 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests3178925015/etcd.yaml"
[upgrade/staticpods] Renewing etcd-server certificate
I1107 13:11:26.571109    3826 staticpods.go:225] Pod manifest files diff:
@@ -35 +35 @@
-    image: registry.k8s.io/etcd:3.5.9-0
+    image: registry.k8s.io/etcd:3.5.10-0