kubernetes: Persistent Volume does not mount due to missing symlink

Is this a BUG REPORT or FEATURE REQUEST?: bug report

/kind bug

What happened: Pod with EBS backed Persistent Volume stays in container creating state and fails to mount pv.

What you expected to happen: Pod should be in running state after a minute or so

How to reproduce it (as minimally and precisely as possible): created 1.7.4 cluster with hyperkube running Linux 4.9.58-18.55.amzn1.x86_64 on AWS created pod with persistent volume

apiVersion: v1
kind: Pod
metadata:
  name: aws-web
spec:
  containers:
    - name: web
      image: nginx
      ports:
        - name: web
          containerPort: 80
          protocol: TCP
      volumeMounts:
        - name: html-volume
 
          mountPath: "/usr/share/nginx/html"
  volumes:
    - name: html-volume
      awsElasticBlockStore: 
        volumeID: vol-0b7063eecf56f008d
        fsType: ext4

Pod stays in container creating.

running a kubectl describe shows this after about 1 minute

Warning  FailedMount            1m    kubelet, ip-172-16-10-202.ec2.internal  Unable to mount volumes for pod "aws-web_default(53ebddff-2871-11e8-aa04-0ab25ab33f88)": timeout expired waiting for volumes to attach/mount for pod "default"/"aws-web". list of unattached/unmounted volumes=[html-volume]

Looking futher into control-manager at -vv 16 I see

aws_util.go:232] nvme path not found "/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0a3923cc3ab6340a9"

It looks like control-manager expects this symlink to exist when it doesn’t.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.7.4
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): NAME=“Amazon Linux AMI” VERSION=“2017.09” ID=“amzn” ID_LIKE=“rhel fedora” VERSION_ID=“2017.09” PRETTY_NAME=“Amazon Linux AMI 2017.09” ANSI_COLOR=“0;33” CPE_NAME=“cpe:/o:amazon:linux:2017.09:ga” HOME_URL=“http://aws.amazon.com/amazon-linux-ami/”
• Kernel (e.g. uname -a): 4.9.58-18.55.amzn1.x86_64
• Install tools: hyperkube
• Others: