kubernetes: Scalability tests failing with "unable to get full preferred group-version-resource" errors

After reverting switching default of dns back to kube-dns - see https://github.com/kubernetes/kubernetes/issues/68613 we recovered in terms of latency metrics, apiserver crashes, etc.

However, in the last run: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/213 [where all metrics etc. were fine], we observed a different problem.

The problem that we observed was that 20 requests failed with the error like this:

I0914 13:02:08.106] Sep 14 13:02:08.106: INFO: Unexpected error occurred: error while scaling RC load-small-6465 to 4 replicas: Error while scaling load-small-6465 to 4 replicas: unable to get client for replicationcontrollers: unable to get full preferred group-version-resource for replicationcontrollers: the cache has not been filled yet
...
I0914 13:58:00.450] Sep 14 13:58:00.449: INFO: Unexpected error occurred: error while scaling RC load-small-7285 to 5 replicas: Error while scaling load-small-7285 to 5 replicas: unable to get client for replicationcontrollers: unable to get full preferred group-version-resource for replicationcontrollers: the cache has not been filled yet

[And those 20 errors were spread over ~1h]

Which is pretty strange, since I don’t know how this error can happen in apiserver initialized 5 hours before. @kubernetes/sig-api-machinery-bugs @liggitt - any thoughts?