kubeadm: Kubeadm fails to init a master with a cloud-config because apiserver can't read it

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

the api server fails with the

I1004 18:20:37.003327       1 server.go:114] Version: v1.8.0
I1004 18:20:37.003445       1 cloudprovider.go:59] --external-hostname was not specified. Trying to get it from the cloud provider.
F1004 18:20:37.003478       1 plugins.go:115] Couldn't open cloud provider configuration /etc/kubernetes/cloud-config: &os.PathError{Op:"open", Path:"/etc/kubernetes/cloud-config", Err:0x2}

the cloud config is there on a host but it seems that container can’t read it.

SELinux is off:

root@kube-master-1:~# getenforce
Disabled

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:46:41Z", GoVersion:"go1.8.3", Compil
er:"gc", Platform:"linux/amd64"}

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler
:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Kernel (e.g. uname -a):

Linux kube-master-1 4.11.0-1011-azure #11-Ubuntu SMP Tue Sep 19 19:03:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Others:

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 10
Comments: 32 (4 by maintainers)

Commits related to this issue

Merge pull request #49840 from andrewrynhard/variable_certs_dir Automatic merge from submit-queue (batch tested with PRs 49840, 54937, 54543). If you want to cherry-pick this change to another branch... — committed to kubernetes/kubernetes by deleted user 7 years ago
Add support for HostPath mounts in Kubeadm (#13) ref: https://github.com/kubernetes/kubeadm/issues/484 — committed to pharmer/pharmer by tamalsaha 7 years ago
Add support for HostPath mounts in Kubeadm (#13) (#225) ref: https://github.com/kubernetes/kubeadm/issues/484 — committed to pharmer/pharmer by tamalsaha 7 years ago
Added a note about mounting vsphere.conf into system containers The default description misses the fact that since Kubernetes version 1.9 additional config files in /etc/kubernetes path should be exp... — committed to artazar/vsphere-storage-for-kubernetes by artazar 5 years ago

Most upvoted comments

@alexpekurovsky just moved from 1.8 to 1.9.5. Is this the correct setup??

Add following to /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Environment="KUBELET_EXTRA_ARGS=--cloud-provider=vsphere --cloud-config=/etc/kubernetes/cloud-config/vsphere.conf"

Update /etc/kubernetes/manifests/kube-apiserver.yaml and /etc/kubernetes/manifests/kube-controller-manager.yaml with

    - --cloud-config=/etc/kubernetes/cloud-config/vsphere.conf
    - --cloud-provider=vsphere

and add new volumeMount

    volumeMounts:
    - mountPath: /etc/kubernetes/cloud-config
      name: cloud-config
      readOnly: true

dkirrane on Mar 20, 2018

I copied my vsphere config to /etc/kubernetes/pki, from initially having it at /etc/kubernetes, then I reloaded my configuration with (sudo systemctl daemon-reload; sudo systemctl restart kubelet.service). Everything worked after that.

Here’s what I have in my manifests:

grep cloud /etc/kubernetes/manifests/kube-apiserver.yaml
    - --cloud-provider=vsphere
    - --cloud-config=/etc/kubernetes/pki/vsphere.conf
grep cloud /etc/kubernetes/manifests/kube-controller-manager.yaml
    - --cloud-provider=vsphere
    - --cloud-config=/etc/kubernetes/pki/vsphere.conf

I’m not exactly sure what you’re asking. Let me know if I didn’t understand. I’m happy to do whatever I can to help track this down if possible - just still n00b status in Kube and learning so apologies if I missed something.

rayterrill on Dec 12, 2017

@luxas What about other optional mounts like for basic auth? You mentioned there was a fix, but I didn’t find it yet.

mattymo on Oct 13, 2017

What’s the correct way to do this for Kubeadm 1.8.7. Is the solution to use /etc/kubernetes/cloud-config/vsphere.conf or still use the workaround /etc/kubernetes/pki/vsphere.conf. What about the other flag --cloud-provider=vsphere?

dkirrane on Feb 12, 2018

Running into this same issue trying to get the vSphere plugin working on a cluster built with kubeadm. Temporarily moving my vsphere config file into the pki directory worked for me as far as allowing the cluster to start again after making the --cloud-provider and --cloud-config settings.

rayterrill on Nov 27, 2017

@tamalsaha That PR will be backported to v1.8.x I think, yes

luxas on Nov 9, 2017

1.8.1 is released. Is this fix included?

sigxcpu76 on Oct 15, 2017