kubernetes: Cluster Deployment Issue with OpenStack Heat (salt://kubelet/ca.crt not found)

Kubernetes version (use kubectl version): Kubernetes release: v1.6.0-beta.3 (same behavior with the previous releases)

Environment:

• Cloud provider or hardware configuration: OpenStack Heat

• OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS (client machine)

• Kernel (e.g. uname -a): Linux control 4.4.0-36-generic

What happened: I’m trying to deploy a kubernetes cluster on OpenStack (kube-up.sh), using the official tutorial (https://kubernetes.io/docs/getting-started-guides/openstack-heat/).

Unfortunately I’m not able connect to kubernetes-master. I’ve checked the logs (/var/log/salt/minion) and the salt-minion reports the error:

2017-03-20 15:51:17,783 [salt.loaded.int.states.file][WARNING ][9745] State for file: /var/log/kube-apiserver.log - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
2017-03-20 15:51:17,784 [salt.loaded.int.states.file][WARNING ][9745] State for file: /var/log/kube-apiserver-audit.log - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.

2017-03-20 15:51:53,249 [salt.state       ][ERROR   ][9745] Source file salt://kubelet/ca.crt not found

2017-03-20 15:51:53,263 [salt.loaded.int.states.file][WARNING ][9745] State for file: /var/log/kube-controller-manager.log - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
2017-03-20 15:51:53,272 [salt.loaded.int.states.file][WARNING ][9745] State for file: /var/log/kube-scheduler.log - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
2017-03-20 15:51:58,981 [py.warnings      ][WARNING ][10939] /usr/lib/python2.7/site-packages/salt/states/cmd.py:1044: DeprecationWarning: The legacy user/group arguments are deprecated. Replace them with runas. These arguments will be removed in Salt Oxygen.

The salt directory /srv/salt/kubelet/ has no ca.crt, so i tried to use the ca.crt from directory /srv/kubernetes/ and salt works, but the service /usr/local/bin/kubelet report an authentication error with openstack (error: failed to run Kubelet: could not init cloud provider “openstack”: Authentication failed).

How to reproduce it (as minimally and precisely as possible):

source DemoProjekt-openrc.sh # containing the credentials for OpenStack
export EXTERNAL_NETWORK="External"
export CREATE_IMAGE=false
export IMAGE_ID=cf47efb4-046e-4a01-9efa-0e5edfde4f1d
export KUBERNETES_PROVIDER=openstack-heat; ./cluster/kube-up.sh
ubuntu@control:~/kubernetes$ ./cluster/kube-up.sh
... Starting cluster using provider: openstack-heat
... calling verify-prereqs
swift client installed
glance client installed
nova client installed
heat client installed
openstack client installed
... calling verify-kube-binaries
... calling kube-up
kube-up for provider openstack-heat
[INFO] Execute commands to create Kubernetes cluster
[INFO] Uploading kubernetes-server-linux-amd64.tar.gz
kubernetes-server.tar.gz
[INFO] Uploading kubernetes-salt.tar.gz
kubernetes-salt.tar.gz
/usr/local/lib/python2.7/dist-packages/novaclient/client.py:278: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
  warnings.warn(msg)
[INFO] Key pair already exists
Stack not found: kube-stack
[INFO] Create stack kube-stack
+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| Field               | Value                                                                                                                                   |
+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| id                  | 628f4d30-45ce-45b1-bb80-87cbd1fa4732                                                                                                    |
| stack_name          | kube-stack                                                                                                                              |
| description         | Kubernetes cluster with one master and one or more worker nodes (as specified by the number_of_minions parameter, which defaults to 3). |
|                     |                                                                                                                                         |
| creation_time       | 2017-03-20T15:48:26                                                                                                                     |
| updated_time        | None                                                                                                                                    |
| stack_status        | CREATE_IN_PROGRESS                                                                                                                      |
| stack_status_reason |                                                                                                                                         |
+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
... calling validate-cluster
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_IN_PROGRESS
Cluster status CREATE_COMPLETE
/usr/local/lib/python2.7/dist-packages/novaclient/client.py:278: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
  warnings.warn(msg)
Cluster "openstack-kube-stack" set.
User "openstack-kube-stack" set.
Context "openstack-kube-stack" set.
Switched to context "openstack-kube-stack".
Wrote config for openstack-kube-stack to /home/ubuntu/.kube/config
Done, listing cluster services:

The connection to the server xxx.xxx.7.114 was refused - did you specify the right host or port?

Anything else we need to know:

The cloud config inside the cluster is correct:
[minion@kube-stack-master ~]$ cat /srv/kubernetes/openstack.conf
[Global]
auth-url=https://openstack:13000/v2.0
username=name
password=password
region=regionOne
tenant-name=DemoProjekt
domain-name=
[LoadBalancer]
lb-version=
subnet-id=a5222d9b-8f31-4b75-8c59-007ea336a6a3
floating-network-id=613a4a6f-f7a0-4a55-7aa7-bffd5d01a915
[Route]
router-id=c950e702-eb37-49c4-a11d-bad0633a4626