kubernetes: Wrong AWS volume can be mounted

Background: K8s often has problems attaching/detaching AWS volumes: If I do a kubectl replace on a deployment, I frequently observe K8s not correctly detaching the volume, and the new deployment will be forever stuck in a crash loop because it times out getting the new volume. Seems like a race condition. The steps I always do when this happens is to delete the deployment, manually force-detach via EC2 API, and manually umount the mounts, otherwise they never seem to be cleaned up.

So today I experienced this again, and I did the following:

1. Delete deployment.
2. Manually detach volume via EC2 API, since it was not being released.
3. Manually umount wrong volume. I accidentally unmounted the volume of a different pod because I didn’t read the mount output correctly. Oops.
4. Created deployment.
5. Noticed my mistake.

What happened was that the new deployment was mounted with the wrong device:

$ mount | grep xvdb
/dev/xvdba on /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-0e3bf8343563ee5aa type ext4 (rw,relatime,data=ordered)
/dev/xvdba on /var/lib/kubelet/pods/a8f54bf6-4ebd-11e6-89fc-065b459e106f/volumes/kubernetes.io~aws-ebs/data type ext4 (rw,relatime,data=ordered)

This is the right volume ID, but it has mounted the wrong device. According to EC2, the (right) volume is attached to /dev/xvdbb.

$ aws ec2 describe-volumes --volume-ids vol-0e3bf8343563ee5aa
{
    "Volumes": [
        {
            "AvailabilityZone": "us-east-1e", 
            "Attachments": [
                {
                    "AttachTime": "2016-07-20T21:05:17.000Z", 
                    "InstanceId": "i-06d03923e9e5a6329", 
                    "VolumeId": "vol-0e3bf8343563ee5aa", 
                    "State": "attached", 
                    "DeleteOnTermination": false, 
                    "Device": "/dev/xvdbb"
                }
            ], 
            "Tags": [],
            "Encrypted": false, 
            "VolumeType": "gp2", 
            "VolumeId": "vol-0e3bf8343563ee5aa", 
            "State": "in-use", 
            "Iops": 100, 
            "SnapshotId": "", 
            "CreateTime": "2016-07-19T00:25:00.168Z", 
            "Size": 30
        }
    ]
}

/dev/xvdba is the volume I accidentally unmounted by mistake:

            // ...
            "Attachments": [
                {
                    "AttachTime": "2016-07-19T17:56:27.000Z", 
                    "InstanceId": "i-06d03923e9e5a6329", 
                    "VolumeId": "vol-0e0a13f1b3a4a6dbc", 
                    "State": "attached", 
                    "DeleteOnTermination": false, 
                    "Device": "/dev/xvdba"
                }
            ], 

So it’s mounting the wrong volume into the pod.

Of course I was being naughty, but it seems to me that this should never happen, since EC2 has already allocated xvdbb?

Here’s the entire KCM output from this fiasco.