kops: Channels is not applying updated addon manifests on existing clusters

1. What kops version are you running? The command kops version, will display this information.

Version 1.10.0 (git-8b52ea6d1)

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-18T11:37:06Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

Create a cluster, then make a change to the network provider’s configuration in the cluster manifest.

Example changes:

apiVersion: kops/v1alpha2
kind: Cluster
spec:
  ...
  networking:
    amazonvpc:
      # uncomment this after the cluster has been created
      # imageName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:1.1.0

or

apiVersion: kops/v1alpha2
kind: Cluster
spec:
  ...
  networking:
    calico:
      # uncomment this after the cluster has been created
      # logSeverityScreen: WARNING
      crossSubnet: true
      prometheusMetricsEnabled: true

Apply the changes:

kops update cluster --yes kops rolling-update cluster --yes --force

5. What happened after the commands executed?

The instance groups were rolled however the network provider’s configuration was not updated.

6. What did you expect to happen?

The networking provider’s DaemonSet should have been updated with the new configuration change.

9. Anything else do we need to know?

I’m not sure if this is an issue in protokube or channels.

Here is what I believe is the relevant portion of the protokube logs:

{"log":"I0823 17:56:17.347666    1830 channels.go:31] checking channel: \"s3://KOPS_STATE_STORE/CLUSTER_NAME/addons/bootstrap-channel.yaml\"\n","stream":"stderr","time":"2018-08-23T17:56:17.368337068Z"}
{"log":"I0823 17:56:17.347734    1830 channels.go:45] Running command: channels apply channel s3://KOPS_STATE_STORE/CLUSTER_NAME/addons/bootstrap-channel.yaml --v=4 --yes\n","stream":"stderr","time":"2018-08-23T17:56:17.36836546Z"}
{"log":"I0823 17:56:17.689504    1830 channels.go:34] apply channel output was: I0823 17:56:17.377322    8747 addons.go:38] Loading addons channel from \"s3://KOPS_STATE_STORE/CLUSTER_NAME/addons/bootstrap-channel.yaml\"\n","stream":"stderr","time":"2018-08-23T17:56:17.692491816Z"}

I exec’ed into the protokube container and reran that channels command and it reported that no update was required even though the daemonset is still out of date. I confirmed that the manifest at s3://KOPS_STATE_STORE/CLUSTER_NAME/addons/networking.amazon-vpc-routed-eni/k8s-1.7.yaml has the updated imageName from the first example in 4., but it seems that channels is not applying it.

Related: #4348 #5055