kubernetes: error message on storage transform failure does not indicate actual cause of the problem

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:

I often get Invalid padding on input whenever I try to start Pods or delete secrets. This doesn’t happen all of the time, however.

They look something like this:

$: kubectl run busybox --image=busybox --command -- sleep 3600
deployment.apps "busybox" created
# Wait a minute or two.
$: pod_name=$(kubectl get pods -l run=busybox -o jsonpath="{.items[0].metadata.name}")
$: kubectl describe pods $pod_name | grep -A 20 "Events"
Events:
  Type     Reason       Age                 From                    Message
  ----     ------       ----                ----                    -------
  Normal   Scheduled    40m                 default-scheduler       Successfully assigned busybox-68654f944b-h92hg to ip-10-0-4-219
  Warning  FailedMount  10m (x23 over 40m)  kubelet, ip-10-0-4-219  MountVolume.SetUp failed for volume "default-token-xqb2r" : Internal error occurred: invalid padding on input
  Warning  FailedMount  13s (x18 over 38m)  kubelet, ip-10-0-4-219  Unable to mount volumes for pod "busybox-68654f944b-h92hg_default(24feb300-952a-11e8-a3b2-0ec4bc8258b0)": timeout expired waiting for volumes to attach or mount for pod "default"/"busybox-68654f944b-h92hg". list of unmounted volumes=[default-token-xqb2r]. list of unattached volumes=[default-token-xqb2r]

What you expected to happen:

I expected containers for the Pod to initialize successfully.

How to reproduce it (as minimally and precisely as possible):

1. Create an encryption token: token=$(cat /dev/urandom | head -c 32 | base64)
2. Put it in this encryption-config.yaml:

cat >/tmp/encryption-config.yaml <<ENCRYPTION_CONFIG
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: $token
      - identity: {}
ENCRYPTION_CONFIG
3. Start the `kube-apiserver` with `--experimental-encryption-provider-config=/tmp/encryption-config.yaml`
4. Attach a few kubelets to it, or restart any bound Kubelets
5. Try to create the `busybox` Pod per above.

Anything else we need to know?:

1. I’m on AWS. I’m happy to share any configurations required to help debug this problem.
2. My original encryption-config.yaml looked like this (I created it per kubernetes-the-hard-way):

kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: OTEnuvc77R2ehn+Zhlwv1jk+UFVZjl7B3Qw+CWMLzMc=
      - identity: {}

Environment:

• Kubernetes version (use kubectl version): 1.10.2
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Ubuntu 18.04 LTS
• Kernel (e.g. uname -a): Linux ip-10-0-1-42 4.15.0-1010-aws #10-Ubuntu SMP Thu May 24 08:41:34 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: none
• Others: