kubernetes: [1.9] kube-controller-manager fails to start with cloud-provider=aws

Solution

Thanks @dims for guiding me the right way.

After looking into the tickets and read the comments by @kingdonb

• https://github.com/kubernetes/kubeadm/issues/90
• https://github.com/kubernetes/kubernetes/issues/37844#issuecomment-286120247
• https://github.com/kubernetes/kubeadm/issues/416

Found the documentation below. Is this in kubernetes.io and if not … Why!?

K8S AWS Cloud Provider Notes

Steps

Tagged EC2/SG

---
#--------------------------------------------------------------------------------
# [K8S AWS Cloud Provider Notes]
# https://docs.google.com/document/d/17d4qinC_HnIwrK0GHnRlD1FKkTNdN__VO4TH9-EzbIY/edit
# Set a tag on all resources in the form of KubernetesCluster=<cluster name>
#  All instances
#  One and only one SG for each instance should be tagged.
#  - This will be modified as necessary to allow ELBs to access the instance
#--------------------------------------------------------------------------------
- name: "Get list of instances in the {{ ENV_ID }} environment..."
#  ec2_remote_facts:  # Deprecated
  ec2_instance_facts:
    filters:
      "tag:environment": "{{ ENV_ID }}"
      instance-state-name: running
    region:          "{{ ec2_region }}"
    ec2_access_key:  "{{ ec2_access_key }}"
    ec2_secret_key:  "{{ ec2_secret_key }}"
  register: ec2

- name: "Tag EC2 instancess"
  ec2_tag:
    region:               "{{ ec2_region }}"
    resource:             "{{ item.instance_id }}"
    tags:
      environment:        "{{ ENV_ID }}"
      KubernetesCluster:  "{{ K8S_CLUSTER_NAME }}"
    aws_access_key:       "{{ ec2_access_key }}"
    aws_secret_key:       "{{ ec2_secret_key }}"
  with_items:     "{{ ec2.instances }}"

- name: "Get list of SGs in the {{ ENV_ID }} environment..."
  ec2_group_facts:
    filters:
      group_name:         "{{ ec2_security_group }}"
    region:               "{{ ec2_region }}"
    ec2_access_key:       "{{ ec2_access_key }}"
    ec2_secret_key:       "{{ ec2_secret_key }}"
  register: sg

- name: "Tag SG"
  ec2_tag:
    region:               "{{ ec2_region }}"
    resource:             "{{ item.group_id }}"
    tags:
      environment:        "{{ ENV_ID }}"
      KubernetesCluster:  "{{ K8S_CLUSTER_NAME }}"
    aws_access_key:       "{{ ec2_access_key }}"
    aws_secret_key:       "{{ ec2_secret_key }}"
  with_items:             "{{ sg.security_groups }}"

Run kubeadm init --config <kubeadm config> to pass cloud provider.

kubeadmin init --config kubeadmin_conf.yaml

[kubeadm_conf.yaml.j2]
kind: MasterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha1
api:
  advertiseAddress: {{ K8S_ADVERTISE_ADDRESS }}
networking:
  podSubnet:        {{ K8S_SERVICE_ADDRESSES }}
cloudProvider:      {{ K8S_CLOUD_PROVIDER }}

Result

Jan 02 04:48:28 ip-172-31-4-117.us-west-1.compute.internal dockerd-current[8063]: I0102 04:48:28.752141
1 reconciler.go:287] attacherDetacher.AttachVolume started for volume "kuard-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0d275986ce24f4304") from node "ip-172-3
Jan 02 04:48:39 ip-172-31-4-117.us-west-1.compute.internal dockerd-current[8063]: I0102 04:48:39.309178
1 operation_generator.go:308] AttachVolume.Attach succeeded for volume "kuard-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0d275986ce24f4304") from node "ip-172-

$ kubectl describe pod kuard
...
Volumes:
  kuard-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  kuard-pvc
    ReadOnly:   false

$ kubectl describe pv kuard-pv
Name:            kuard-pv
Labels:          failure-domain.beta.kubernetes.io/region=us-west-1
                 failure-domain.beta.kubernetes.io/zone=us-west-1b
                 type=amazonEBS
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"type":"amazonEBS"},"name":"kuard-pv","namespace":""},"spec":{"acce...
                 pv.kubernetes.io/bound-by-controller=yes
StorageClass:    
Status:          Bound
Claim:           default/kuard-pvc
Reclaim Policy:  Retain
Access Modes:    RWO
Capacity:        5Gi
Message:         
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   vol-0d275986ce24f4304
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>

$ kubectl version -o json
{
  "clientVersion": {
    "major": "1",
    "minor": "9",
    "gitVersion": "v1.9.0",
    "gitCommit": "925c127ec6b946659ad0fd596fa959be43f0cc05",
    "gitTreeState": "clean",
    "buildDate": "2017-12-15T21:07:38Z",
    "goVersion": "go1.9.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "serverVersion": {
    "major": "1",
    "minor": "9",
    "gitVersion": "v1.9.0",
    "gitCommit": "925c127ec6b946659ad0fd596fa959be43f0cc05",
    "gitTreeState": "clean",
    "buildDate": "2017-12-15T20:55:30Z",
    "goVersion": "go1.9.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}

Hi @dims, I have read your comments. Still the point is why --cloud-provider option does not work as it is clearly documented in the kube-controller-manager reference document.

--cloud-provider string                                             The provider for cloud services.  Empty string for no provider.

If this option does not work, then need to clarify if it is as expected or not. If it is as expected, then the documentation fix would be raised, if it is not, then need to clarify why.

If this issue still need to be closed, I suppose all related issues like below would need to be closed too and make it clear that giving --cloud-provider option is not to use but need to give it to kubeadm only if kubeadm is used.

• https://github.com/kubernetes/kubernetes/issues/55048
• https://github.com/kubernetes/kubernetes/issues/56610

Please give a clarification and the mechanism behind why giving --cloud-provider option to kube-controller-manager does not work although is clearly documented.

Possible suggestion would be re-run kubeadm init after kubeadm reset. However it is also problematic and needs manual clean up of CNI files, etc which will prevent kubeadm init to run.

Initially giving the --cloud-provider=aws in systemd unit file for kubelet stopped the error in kubelet journelctl output, but it did not stop the errors in kube-controller-manager. If we can simply provide --cloud-provider=aws to kube-controller-manager, it would be a good way for all of us who get into this --cloud-provider related issues. Also api-server can take the option with no issue. So why kube-controller-manager fails and what is specific with kube-controller-manager.

For all reading this issue. the following comment helped me alot!

https://github.com/kubernetes/kubernetes/issues/53538#issuecomment-345942305

Hi @dims. I am running into a similar issue. Just a couple of questions/confusions on your and @oonisim 's provided solutions:

Set a tag on all resources in the form of KubernetesCluster=<cluster name> All instances One and only one SG for each instance should be tagged. This will be modified as necessary to allow ELBs to access the instance

Is it really necessary to tag the resources? (Just asking to make things simpler)If so, then where can I find the <cluster name>?

Specifying --config with kubeadm init with the following contents

Does this config file need to be created? Or is it located somwhere? If so, where?

Making sure /etc/kubernetes/cloud-config is created in each node including master with the correct information needed for the specific cloud provider

Can you please specify exactly what should be the content of the cloud-config for AWS?

1. Lastly, is it possible to modify the cloud-provider of the existing cluster without doing kubeadm init again and specifying the kubeadm.conf. Because what I understand right now is that we need to reset the cluster and then kubeadm init again with the kubeadm.conf to change the cloud-provider. Please correct me if I am wrong.

Thanks

Hi @dims, why this is closed? the original issue of not being able to specify --cloud-provider=aws in /etc/kubernetes/manifest/kube-controller-manager.yaml is still there. Please clarify if it is the expected behaviour, or otherwise please reopen to clarify why it does not work.

The installation guide Using kubeadm to Create a Cluster does not mention Cloud Provider and it needs to specify --cloud-provider or configuration file including cloud provider with --config. It would not be uncommon later people notice cloud provider configuration is required and try to reconfigure API server, kubelet, Controller.

In kubeadm join unknown flag: --cloud-provider it says:

You should just edit /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and it will work

If to add --cloud-provider=aws in the config file is the way for kubelet, then it should be the same for API server and controller.