kubernetes: Running 1.13.x kubelet against 1.12.x apiserver errors due to missing volumeMode

Upgrading a cluster from 1.12.3 to 1.13.0 causes pods to break because the “volumeMode” is missing.

What happened:

all pods with volumes became unavailable:

Dec 6 09:31:46 experience kubelet[24011]: E1206 09:31:46.522733 24011 desired_state_of_world_populator.go:296] Error processing volume "media" for pod "djangofluent-tst-test-6cfc6555-9bfm6_fluentdemo(0dd95f6c-ed7e-11e8-afe8-5254000919ee)": cannot get volumeMode for volume: djangofluent-tst-media

Dec 6 09:31:46 experience kubelet[24011]: E1206 09:31:46.919223 24011 desired_state_of_world_populator.go:296] Error processing volume "media" for pod "djangofluent-prd-production-7c765b5c58-6kprb_fluentdemo(0dd8554c-ed7e-11e8-afe8-5254000919ee)": cannot get volumeMode for volume: djangofluent-prd-media

Dec 6 09:47:38 experience kubelet[1926]: E1206 09:47:38.027881 1926 desired_state_of_world_populator.go:296] Error processing volume "redis-data" for pod "redis-master-0_infra(eb93df30-ed7d-11e8-afe8-5254000919ee)": cannot get volumeMode for volume: redis-data-redis-master-0

What you expected to happen:

Kubelet would default to FileSystem when the volumeMode is not present.

How to reproduce it (as minimally and precisely as possible):

• Run a kubernetes 1.12.3 cluster (installed bare metal with kubeadm).
• apt-get dist-upgrade for kubelet, kubeadm, kubectl
• systemctl restart kubelet

Anything else we need to know?:

Environment:

• Kubernetes version: 1.12.3 -> 1.13.0
• OS (e.g. from /etc/os-release): Debian 9.6
• Install tools: kubeadm on bare metal

/kind documentation