kubeadm: kubeadm upgrade fails when hostname != node name

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version): kubeadm version: &version.Info{Major:“1”, Minor:“12”, GitVersion:“v1.12.1”, GitCommit:“4ed3216f3ec431b140b1d899130a69fc671678f4”, GitTreeState:“clean”, BuildDate:“2018-10-05T16:43:08Z”, GoVersion:“go1.10.4”, Compiler:“gc”, Platform:“linux/amd64”}

Environment:

Kubernetes version (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“12”, GitVersion:“v1.12.1”, GitCommit:“4ed3216f3ec431b140b1d899130a69fc671678f4”, GitTreeState:“clean”, BuildDate:“2018-10-05T16:46:06Z”, GoVersion:“go1.10.4”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“11”, GitVersion:“v1.11.2”, GitCommit:“bb9ffb1654d4a729bb4cec18ff088eacc153c239”, GitTreeState:“clean”, BuildDate:“2018-08-07T23:08:19Z”, GoVersion:“go1.10.3”, Compiler:“gc”, Platform:“linux/amd64”}
Cloud provider or hardware configuration: KVM
OS (e.g. from /etc/os-release): Container Linux by CoreOS 1855.4.0 (Rhyolite)
Kernel (e.g. uname -a): Linux k8s-m01 4.14.67-coreos #1 SMP Mon Sep 10 23:14:26 UTC 2018 x86_64 Intel Core i7 (Nehalem Class Core i7) GenuineIntel GNU/Linux

What happened?

I am trying to upgrade from Kubernetes from v1.11.2 to v1.12.1 using the command kubeadm upgrade apply v1.12.1. However, this times out:

[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.12.1"... [upgrade/apply] FATAL: timed out waiting for the condition

Looking at the verbose logs, this appears to be because kubeadm is using the hostname to find the apiserver pod, rather than the Kubernetes node name:

I1015 10:59:20.235573 16962 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.12.1 (linux/amd64) kubernetes/4ed3216" 'https://192.168.1.1:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-HOSTNAME' I1015 10:59:20.267945 16962 round_trippers.go:405] GET https://192.168.1.1:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-HOSTNAME 404 Not Found in 32 milliseconds

What you expected to happen?

The cluster is upgraded successfully.

How to reproduce it (as minimally and precisely as possible)?

Upgrade a Kubernetes cluster where the master node has a hostname different to the node name specified when initialising the Kubernetes cluster.

kubeadm init --node-name=NODENAME kubeadm upgrade apply v1.12.1

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 3
Comments: 22 (13 by maintainers)

Most upvoted comments

I hit this today on multiple clusters. After digging a bit I noted that you need to pass the kube-apiserver-HOSTNAME to the InitConfiguration of your kubeadm config, reusing the old configuration will not work and doing a kubeadm config view seems to be broken in this version as per #1174 .

Here’s the configuration I’m using to have this working:

apiEndpoint:
  advertiseAddress: 172.32.10.90
  bindPort: 6443
apiVersion: kubeadm.k8s.io/v1alpha3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: ""
  usages:
  - signing
  - authentication
kind: InitConfiguration
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: ip-172-32-10-90.ec2.internal
  kubeletExtraArgs:
    cloud-provider: "aws"
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master

The code referring to this is here: https://github.com/kubernetes/kubernetes/blob/4ed3216f3ec431b140b1d899130a69fc671678f4/cmd/kubeadm/app/phases/upgrade/staticpods.go#L394

fntlnz on Oct 17, 2018

@rdodev - can you reproduce this?

timothysc on Oct 31, 2018

hi @yagonobre

https://github.com/kubernetes/kubernetes/issues/61664 the --hostname-override flag for the kubelet is going away eventually so we don’t want look it up in /var/lib/kubelet/kubeadm-flags.env and don’t want to bind more logic to that file.

one way to do this, is to query the kubelet.conf file in “etc/kubernetes” which is a Config type and from there look at the users and contexts.
another way, for a running kube-system pod (e.g. api-server) is to get it’s podspec and look up nodeName.

if this doesn’t get more comments within 2-3 working days just give 1) a go with a PR.

/cc @kubernetes/sig-cluster-lifecycle opinions on 1 vs 2, or perhaps use another method?

neolit123 on Oct 19, 2018