spinnaker: Using Halyard, adding Kubernetes accounts in clouddriver-local.yml causes clouddriver-caching (and -rw) to throw "error: stat /home/userid/.kube/config: no such file or directory" after install

Issue Summary:

clouddriver-rw and clouddriver-caching services work fine if the Kubernetes accounts are defined as part of ~/.hal/config. However, customizing the services using ~/.hal/default/profiles/clouddriver-local.yml files fails with:

error: stat /home/userid/.kube/config: no such file or directory

Cloud Provider(s):

Kubernetes 1.11.2 installed using kubeadm

Environment:

Spinnaker 1.10.0

Feature Area:

Clouddriver

Description:

I was actually trying to test the new checkPermissionsOnStartup feature included in 1.10 as documented in the issue #3486

In the process, I noticed that none of my Kubernetes accounts would load as neither clouddriver-rw or clouddriver-caching could find the kube config file. What’s surprising is, we do not see this issue if the exact same configuration is inside ~/.hal/config file

Steps to Reproduce:

1. Setup a simplistic hal config file with 1 Kubernetes v2 account. Define the account details in ~/.hal/config file directly
2. Run hal deploy apply. Things should come up with no issues
3. Now, take the Kubernetes v2 account definitions and move it to clouddriver-local.yml
4. Both clouddriver-rw and clouddriver-caching will start failing with “error: stat /home/<userid>/.kube/config: no such file or directory”

Additional Details:

Here’s the reason why defining Kubernetes v2 accounts in clouddriver-local.yml file leads to the error that I reported above:

When you do not use clouddriver-local.yml and define all your accounts in halyard, halyard sets up the following things correctly when it comes to deploying clouddriver-* microservices:

• volumeMounts the secret spin-clouddriver-rw-files-[0-9]+ to /opt/spinnaker/config. This secret/volume contains, among other things, clouddriver.yml file
• volumeMounts the secret spin-clouddriver-rw-files-[0-9]+ to /home/ubuntu/.hal/default/staging/dependencies. This secret/volume contains, among other things, [0-9]+-config file which contains all the Kubernetes Config File entries
• hal takes the Kubernetes v2 account entries defined inside ~/.hal/config, copies it to the clouddriver.yml secret data entry, but updates the following key with the appropriate volumeMount path similar to this: kubeconfigFile: /home/ubuntu/.hal/default/staging/dependencies/[0-9]+-config. This step is what ensures that when clouddriver-* microservices come up, they can find all the kubeconfig entries

However, when you define your accounts inside clouddriver-local.yml as opposed to ~/.hal/config, halyard does the following while deploying clouddriver-* microservices:

• volumeMounts the secret spin-clouddriver-rw-files-[0-9]+ to /opt/spinnaker/config. This secret/volume contains, among other things, clouddriver.yml file
• volumeMounts the secret spin-clouddriver-rw-files-[0-9]+ to /home/ubuntu/.hal/default/staging/dependencies. This secret/volume contains, among other things, [0-9]+-config file which contains all the Kubernetes Config File entries
• hal takes the Kubernetes v2 account entries defined inside ~/.hal/config, copies it to the clouddriver.yml secret data entry, but updates the following key with the appropriate volumeMount path similar to this: kubeconfigFile: /home/ubuntu/.hal/default/staging/dependencies/[0-9]+-config. However, hal does not perform the same swapping of kubeconfigFile value with the clouddriver-local.yml secret data entry. It leaves it as kubeconfigFile: /home/ubuntu/.kube/config. Since values inside clouddriver-local.yml overwrites the values inside clouddriver.yml, this kubeconfigFile value causes clouddriver-rw and clouddriver-caching to throw the following error in the logs

error: stat /home/userid/.kube/config: no such file or directory

clouddriver.yml inside /opt/spinnaker/config

clouddriver-local.yml inside /opt/spinnaker/config