kubernetes: kube-apiserver --storage-backend defaults to etcd3 and hangs if connecting to etcd2 cluster

I just spent a lot of time debugging this so I figured I’d raise, if anything just to help the next guy googling why his upgrade from k8s-1.5 to k8s-1.6 isn’t working. But k8s maintainers feel free to close this if there’s no action needed here.

It looks like kube-apiserver for k8s 1.6 now defaults to --storage-backend=etcd3. When in this mode, kube-apiserver hangs on its requests to an etcdv2 server and doesn’t provide much information on the nature of the issue. I saw the following kube-apiserver logs:

9462 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *api.Secret: Get http://localhost:8080/api/v1/secrets?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused

k8s-master kube-apiserver[9462]: E0324 19:05:29.593771    9462 storage_rbac.go:140] unable to initialize clusterroles: the server cannot complete the requested operation at this time, try again later (get clusterroles.rbac.authorization.k8s.io)

storage_extensions.go:127] third party resource sync failed: the server cannot complete the requested operation at this time, try again later (get thirdpartyresources.extensions)
reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *api.Secret: the server cannot complete the requested operation at this time, try again later (get secrets)
9462 storage_rbac.go:140] unable to initialize clusterroles: the server cannot complete the requested operation at this time, try again later (get clusterroles.rbac.authorization.k8s.io)

kubectl requests simply hung, even though the apiserver was responding:

 curl localhost:8080/version
{
  "major": "1",
  "minor": "6+",
  "gitVersion": "v1.6.0-beta.4",
  "gitCommit": "b202120be3a97e5f8a5e20da51d0b6f5a1eebd31",
  "gitTreeState": "clean",
  "buildDate": "2017-03-17T17:42:22Z",
  "goVersion": "go1.7.5",
  "compiler": "gc",
  "platform": "linux/amd64"
}

and there was no indication of any issues in the etcd logs.

Manually setting to --storage-backend=etcd2 resolved my issue.