kubernetes: kube-controller-manager: error creating self-signed certificates, permission denied on v1.12.0-rc.2

BUG REPORT:

/kind bug

What happened:

Kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes (i.e. apiserver, controller-manager, scheduler run as pods) that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops.

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY     STATUS             RESTARTS   AGE
kube-system   kube-apiserver-2t77v                       1/1       Running            2          8m
kube-system   kube-controller-manager-6d6cf5968c-r4jzv   0/1       CrashLoopBackOff   6          9m
kube-system   kube-controller-manager-6d6cf5968c-szd6z   0/1       CrashLoopBackOff   6          9m
...

$ kubectl logs kube-controller-manager-6d6cf5968c-r4jzv -n kube-system
error creating self-signed certificates: mkdir /var/run/kubernetes: permission denied

Notably, the kube-controller-manager is run with:

     securityContext:
        runAsNonRoot: true
        runAsUser: 65534

which has been fine through to v1.11.3. Full kube-controller-manager.yaml.

What you expected to happen:

Controller manager may not assume it has root. Or should clarify if an additional mount is now required.

May have been introduced in https://github.com/kubernetes/kubernetes/pull/65867 cc @sttts

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.12.0-rc.2
• Cloud provider or hardware configuration: GCE, DigitalOcean, AWS
• OS (e.g. from /etc/os-release): Container Linux 1855.4.0
• Kernel (e.g. uname -a): 4.14.67-coreos
• Install tools: Typhoon