kubeadm: Upgrading from 1.9.6 to 1.10.0 fails with timeout

Temporary workaround is to ensure certs and upgrade the etcd and apiserver pods by bypassing the checks.

Be sure to check your Config and add any flags for your use case:

kubectl -n kube-system edit cm kubeadm-config  # change featureFlags
...
  featureGates: {}
...
kubeadm alpha phase certs all
kubeadm alpha phase etcd local
kubeadm alpha phase controlplane all
kubeadm alpha phase upload-config

@codepainters I think it is the same issue.

There are a few underlying problems causing this issue:

• The upgrade is generating a hash of the mirror pod for each component from the result of querying the mirror pod from the API. The upgrade then tests to see if this hashed value changes to determine if the pod is updated from the static manifest change. The hashed value includes fields that can be mutated for reasons other than the static manifest change (such as pod status updates). If the pod status changes between the hash comparisons, then the upgrade will continue to the next component prematurely.
• The upgrade performs the etcd static pod manifest update (including adding tls security to etcd) and attempts to use the apiserver to verify that the pod has been updated, however the apiserver manifest has not been updated at this point to use tls for communicating with etcd.

As a result, the upgrade only succeeds currently when there happens to be a pod status update for the etcd pod that causes the hash to change prior to the kubelet picking up the new static manifest for etcd. Additionally, the api server needs to remain available for the first part of the apiserver upgrade when the upgrade tooling is querying the api prior to updating the apiserver manifest.

@renich just give it the filepath of your config

If you don’t use any custom settings, you can pass it an empty file. Here’s a simple way to do that in bash:

1.10_kubernetes/server/bin/kubeadm alpha phase upload-config --config <(echo)

@kvaps @stealthybox this is most likely etcd issue (kubeadm speaks plain HTTP/2 to TLS-enabled etcd), I hit it too. See this other issue: https://github.com/kubernetes/kubeadm/issues/755

Honestly, I can’t understand why is the same TCP port used for both TLS and non-TLS etcd listeners, it only causes troubles like this one. Getting plain, old connection refused would give immediate hint, here I had to resort to tcpdump to understand what’s going on.

This should now be resolved with the merging of https://github.com/kubernetes/kubernetes/pull/62655 and will be part of the v1.10.2 release.

Thanks @stealthybox For me the upgrade apply process stalled on [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.10.1"... however the cluster was successfully upgraded.

PR to address the static pod update race condition: https://github.com/kubernetes/kubernetes/pull/61942 cherry-pick PR for release-1.10 branch: https://github.com/kubernetes/kubernetes/pull/61954

@detiber and I got on a call to discuss changes we need to make to the upgrade process. We plan to implement 3 fixes for this bug in the 1.10.x patch releases:

• Remove etcd TLS from upgrade.
  The current upgrade loop does batch modifications per component in a serial manner.
  Upgrading a component has no knowledge of dependent component configurations.
  Verifying an upgrade requires the APIServer is available to check the pod status.
  Etcd TLS requires a coupled etcd+apiserver configuration change which breaks this contract.
  This is the minimum viable change to fix this issue, and leaves upgraded clusters with insecure etcd.

• Fix the mirror-pod hash race condition on pod status change.
  https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/upgrade/staticpods.go#L189.
  Upgrades will now be correct assuming compatibility between etcd and apiserver flags.

• Upgrade TLS specifically in a separate phase.
  Etcd and the APIServer need to be upgraded together.
  kubeadm alpha phase ensure-etcd-tls ?.
  This phase should be runnable independently of a cluster upgrade.
  During a cluster upgrade, this phase should run before updating all of the components.

For 1.11 we want to:

• Use the kubelet API for runtime checks of upgraded static pods.
  It’s undesirable to rely on the apiserver and etcd for monitoring local processes like we are currently doing.
  A local source of data about pods is superior to relying on higher-order distributed kubernetes components.
  This will replace the current pod runtime checks in the upgrade loop.
  This will allow us to add checks to the ensure-etcd-tls phase.

alternative: Use the CRI to get pod info (demo’d viable using crictl).
caveat: CRI on dockershim and possibly other container runtimes does not currently support backward compatibility for CRI breaking changes.

I just hit another weird edge case related to this bug. The kubeadm upgrade marked the etcd upgrade as complete prior to the new etcd image being pulled and the new static pod being deployed. This causes the upgrade to timeout at a later step and the upgrade rollback to fail. This also leaves the cluster in a broken state. Restoring the original etcd static pod manifest is needed to recover the cluster.

After retrying this for 10 times it finally worked