kubernetes: Failed to add label for pod security policy limt

What happened:

1. After upgrade kubernetes from v1.12.3 to v1.13.4, pod is in round crash, log shows failed to add label for psp limit. $ kubectl logs --v=8 sg-elastic-data-0 -nanalytics I0516 11:49:15.980544 29784 loader.go:359] Config loaded from file /root/.kube/config I0516 11:49:15.981476 29784 loader.go:359] Config loaded from file /root/.kube/config I0516 11:49:15.988848 29784 loader.go:359] Config loaded from file /root/.kube/config I0516 11:49:15.989305 29784 round_trippers.go:416] GET https://10.75.86.67:8443/api/v1/namespaces/analytics/pods/sg-elastic-data-0 I0516 11:49:15.989327 29784 round_trippers.go:423] Request Headers: I0516 11:49:15.989339 29784 round_trippers.go:426] Accept: application/json, / I0516 11:49:15.989351 29784 round_trippers.go:426] User-Agent: kubectl/v1.13.4 (linux/amd64) kubernetes/c27b913 I0516 11:49:16.001251 29784 round_trippers.go:441] Response Status: 200 OK in 11 milliseconds I0516 11:49:16.001279 29784 round_trippers.go:444] Response Headers: I0516 11:49:16.001291 29784 round_trippers.go:447] Content-Type: application/json I0516 11:49:16.001299 29784 round_trippers.go:447] Date: Thu, 16 May 2019 11:49:16 GMT I0516 11:49:16.001404 29784 request.go:942] Response Body: {“kind”:“Pod”,“apiVersion”:“v1”,“metadata”:{“name”:“sg-elastic-data-0”,“generateName”:“sg-elastic-data-”,“namespace”:“analytics”,“selfLink”:“/api/v1/namespaces/analytics/pods/sg-elastic-data-0”,“uid”:“5ba5b67c-763f-11e9-917e-9418820b408c”,“resourceVersion”:“8262746”,“creationTimestamp”:“2019-05-14T11:56:50Z”,“labels”:{“app”:“elasticsearch”,“controller-revision-hash”:“sg-elastic-data-5c8d6dd45”,“node-name”:“sg-elastic-data”,“statefulset.kubernetes.io/pod-name”:“sg-elastic-data-0”},“annotations”:{“kubernetes.io/psp”:“privileged”},“ownerReferences”:[{“apiVersion”:“apps/v1”,“kind”:“StatefulSet”,“name”:“sg-elastic-data”,“uid”:“e9dc168f-7547-11e9-a788-9418820ba198”,“controller”:true,“blockOwnerDeletion”:true}]},“spec”:{“volumes”:[{“name”:“elasticsearch-config”,“configMap”:{“name”:“elasticsearch-config”,“defaultMode”:420}},{“name”:“sg-elastic-data-certificates”,“secret”:{“secretName”:“sg-elastic-data-certificates”,“items”:[{“key”:“truststore.jks”,“path”:“truststore.jks”,“mode”:420},{“key”:"sg-elastic-data-keystore.j [truncated 5793 chars] I0516 11:49:16.007210 29784 loader.go:359] Config loaded from file /root/.kube/config I0516 11:49:16.007569 29784 round_trippers.go:416] GET https://10.75.86.67:8443/api/v1/namespaces/analytics/pods/sg-elastic-data-0/log I0516 11:49:16.007587 29784 round_trippers.go:423] Request Headers: I0516 11:49:16.007596 29784 round_trippers.go:426] Accept: application/json, / I0516 11:49:16.007614 29784 round_trippers.go:426] User-Agent: kubectl/v1.13.4 (linux/amd64) kubernetes/c27b913 I0516 11:49:16.020283 29784 round_trippers.go:441] Response Status: 200 OK in 12 milliseconds I0516 11:49:16.020305 29784 round_trippers.go:444] Response Headers: I0516 11:49:16.020317 29784 round_trippers.go:447] Content-Type: text/plain I0516 11:49:16.020325 29784 round_trippers.go:447] Date: Thu, 16 May 2019 11:49:16 GMT This POD environment variables: PARENT_HOSTNAME=grey-manager-2 ES_IP=10.75.86.68 ES_PORT=9301 HOSTNAME_IP_MAPPING=grey-manager-1 10.75.86.67 grey-manager-2 10.75.86.68 grey-collector-1 10.75.86.69 grey-collector-2 10.75.86.70 ESTYPE_PORT_MAPPING=sg-elastic-data 9301 sg-elastic-logs 9302 sg-elastic-client 9300 Setting POD label parent-hostname: grey-manager-2

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  **"message": "pods \"sg-elastic-data-0\" is forbidden: unable to validate against any pod security policy: [spec.volumes[3].hostPath.pathPrefix: Invalid value: \"/var/esdata/es_data\": is not allowed to be used spec.volumes[4].hostPath.pathPrefix: Invalid value: \"/es_backup\": is not allowed to be used]",
  "reason": "Forbidden",
  "details": {
    "name": "sg-elastic-data-0",
    "kind": "pods"
  },
  "code": 403
}Checking if key was set successfully...
ERROR: Requested key parent-hostname not found:** {
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "sg-elastic-data-0",
    "generateName": "sg-elastic-data-",
    "namespace": "analytics",
    "selfLink": "/api/v1/namespaces/analytics/pods/sg-elastic-data-0",
    "uid": "5ba5b67c-763f-11e9-917e-9418820b408c",
    "resourceVersion": "8262427",
    "creationTimestamp": "2019-05-14T11:56:50Z",
    "labels": {
      "app": "elasticsearch",
      "controller-revision-hash": "sg-elastic-data-5c8d6dd45",
      "node-name": "sg-elastic-data",
      "statefulset.kubernetes.io/pod-name": "sg-elastic-data-0"
    },
    "annotations": {
      "kubernetes.io/psp": "privileged"
    },
    "ownerReferences": [
      {
        "apiVersion": "apps/v1",
        "kind": "StatefulSet",
        "name": "sg-elastic-data",
        "uid": "e9dc168f-7547-11e9-a788-9418820ba198",
        "controller": true,
        "blockOwnerDeletion": true
      }
    ]
  },
  "spec": {
    "volumes": [
      {
        "name": "elasticsearch-config",
        "configMap": {
          "name": "elasticsearch-config",
          "defaultMode": 420
        }
      },
      {
        "name": "sg-elastic-data-certificates",
        "secret": {
          "secretName": "sg-elastic-data-certificates",
          "items": [
            {
              "key": "truststore.jks",
              "path": "truststore.jks",
              "mode": 420
            },
            {
              "key": "sg-elastic-data-keystore.jks",
              "path": "node-keystore.jks",
              "mode": 420
            }
          ],
          "defaultMode": 420
        }
      },
      {
        "name": "elasticsearch-scripts",
        "configMap": {
          "name": "elasticsearch-scripts",
          "items": [
            {
              "key": "distribute-mounted-files.sh",
              "path": "distribute-mounted-files.sh",
              "mode": 448
            },
            {
              "key": "docker-entrypoint.sh",
              "path": "docker-entrypoint.sh",
              "mode": 448
            },
            {
              "key": "start-elasticsearch.sh",
              "path": "start-elasticsearch.sh",
              "mode": 448
            },
            {
              "key": "get-value-from-map.sh",
              "path": "get-value-from-map.sh",
              "mode": 448
            },
            {
              "key": "set-pod-label.sh",
              "path": "set-pod-label.sh",
              "mode": 448
            }
          ],
          "defaultMode": 420
        }
      },
      {
        "name": "elasticsearch-data",
        "hostPath": {
          "path": "/var/esdata/es_data",
          "type": ""
        }
      },
      {
        "name": "elasticsearch-backups",
        "hostPath": {
          "path": "/es_backup",
          "type": ""
        }
      },
      {
        "name": "timezone-config",
        "hostPath": {
          "path": "/etc/localtime",
          "type": ""
        }
      },
      {
        "name": "default-token-zgh7z",
        "secret": {
          "secretName": "default-token-zgh7z",
          "defaultMode": 420
        }
      }
    ],
    "containers": [
      {
        "name": "sg-elastic-data",
        "image": "bcmt-registry:5000/master/elasticsearch:build_3010",
        "command": [
          "/bin/bash",
          "-c"
        ],
        "args": [
          "/volumes/scripts/docker-entrypoint.sh"
        ],
        "ports": [
          {
            "name": "http",
            "containerPort": 9200,
            "protocol": "TCP"
          },
          {
            "name": "transport",
            "containerPort": 9301,
            "protocol": "TCP"
          }
        ],
        "env": [
          {
            "name": "PARENT_HOSTNAME",
            "valueFrom": {
              "fieldRef": {
                "apiVersion": "v1",
                "fieldPath": "spec.nodeName"
              }
            }
          },
          {
            "name": "HOSTNAME_IP_MAPPING",
            "valueFrom": {
              "configMapKeyRef": {
                "name": "mapping-config",
                "key": "hostnameIpMapping"
              }
            }
          },
          {
            "name": "ESTYPE_PORT_MAPPING",
            "valueFrom": {
              "configMapKeyRef": {
                "name": "mapping-config",
                "key": "elasticsearchPortMapping"
              }
            }
          },
          {
            "name": "ES_JAVA_OPTS",
            "value": "-Xms32212254720 -Xmx32212254720 -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:+UnlockDiagnosticVMOptions -XX:+PrintCompressedOopsMode"
          },
          {
            "name": "NODE_NAME",
            "value": "sg-elastic-data"
          },
          {
            "name": "NODE_TAG",
            "value": "data"
          },
          {
            "name": "IS_NODE_CLIENT",
            "value": "false"
          },
          {
            "name": "IS_NODE_MASTER",
            "value": "false"
          },
          {
            "name": "IS_NODE_DATA",
            "value": "true"
          },
          {
            "name": "DISCOVERY_SERVICE",
            "value": "elasticsearch-transport"
          },
          {
            "name": "NAMESPACE",
            "valueFrom": {
              "fieldRef": {
                "apiVersion": "v1",
                "fieldPath": "metadata.namespace"
              }
            }
          },
          {
            "name": "NUMBER_OF_MASTER_NODES",
            "value": "3"
          },
          {
            "name": "TRUSTSTORE_PASSWORD",
            "valueFrom": {
              "secretKeyRef": {
                "name": "elasticsearch-secrets",
                "key": "truststore.password"
              }
            }
          },
          {
            "name": "KEYSTORE_PASSWORD",
            "valueFrom": {
              "secretKeyRef": {
                "name": "elasticsearch-secrets",
                "key": "keystore.data.password"
              }
            }
          },
          {
            "name": "INDEXLISTER_PASSWORD",
            "valueFrom": {
              "secretKeyRef": {
                "name": "elasticsearch-secrets",
                "key": "indexlister.password"
              }
            }
          },
          {
            "name": "NSS_LOCAL_SITE_NAME",
            "value": "grey"
          },
          {
            "name": "CLUSTER_NAME_SUFFIX",
            "value": "-sg-cluster"
          }
        ],
        "resources": {
          "limits": {
            "memory": "64424509440"
          },
          "requests": {
            "memory": "32212254720"
          }
        },
        "volumeMounts": [
          {
            "name": "elasticsearch-config",
            "readOnly": true,
            "mountPath": "/volumes/config"
          },
          {
            "name": "sg-elastic-data-certificates",
            "readOnly": true,
            "mountPath": "/volumes/certificates"
          },
          {
            "name": "elasticsearch-scripts",
            "readOnly": true,
            "mountPath": "/volumes/scripts"
          },
          {
            "name": "elasticsearch-data",
            "mountPath": "/usr/share/elasticsearch/data"
          },
          {
            "name": "elasticsearch-backups",
            "mountPath": "/usr/share/elasticsearch/backups"
          },
          {
            "name": "timezone-config",
            "readOnly": true,
            "mountPath": "/etc/localtime"
          },
          {
            "name": "default-token-zgh7z",
            "readOnly": true,
            "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
          }
        ],
        "livenessProbe": {
          "tcpSocket": {
            "port": 9200
          },
          "initialDelaySeconds": 240,
          "timeoutSeconds": 5,
          "periodSeconds": 5,
          "successThreshold": 1,
          "failureThreshold": 3
        },
        "readinessProbe": {
          "tcpSocket": {
            "port": 9200
          },
          "timeoutSeconds": 5,
          "periodSeconds": 10,
          "successThreshold": 1,
          "failureThreshold": 3
        },
        "terminationMessagePath": "/dev/termination-log",
        "terminationMessagePolicy": "File",
        "imagePullPolicy": "IfNotPresent"
      }
    ],
    "restartPolicy": "Always",
    "terminationGracePeriodSeconds": 30,
    "dnsPolicy": "Default",
    "nodeSelector": {
      "manager": "true"
    },
    "serviceAccountName": "default",
    "serviceAccount": "default",
    "nodeName": "grey-manager-2",
    "securityContext": {
      
    },
    "hostname": "sg-elastic-data-0",
    "subdomain": "elasticsearch-transport",
    "affinity": {
      "podAntiAffinity": {
        "requiredDuringSchedulingIgnoredDuringExecution": [
          {
            "labelSelector": {
              "matchExpressions": [
                {
                  "key": "node-name",
                  "operator": "In",
                  "values": [
                    "sg-elastic-data"
                  ]
                }
              ]
            },
            "topologyKey": "kubernetes.io/hostname"
          }
        ]
      }
    },
    "schedulerName": "default-scheduler",
    "priority": 0,
    "enableServiceLinks": true
  },
  "status": {
    "phase": "Running",
    "conditions": [
      {
        "type": "Initialized",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-05-14T12:09:49Z"
      },
      {
        "type": "Ready",
        "status": "False",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-05-14T12:09:49Z",
        "reason": "ContainersNotReady",
        "message": "containers with unready status: [sg-elastic-data]"
      },
      {
        "type": "ContainersReady",
        "status": "False",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-05-14T12:09:49Z",
        "reason": "ContainersNotReady",
        "message": "containers with unready status: [sg-elastic-data]"
      },
      {
        "type": "PodScheduled",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-05-14T12:09:49Z"
      }
    ],
    "hostIP": "10.75.86.68",
    "podIP": "172.20.2.18",
    "startTime": "2019-05-14T12:09:49Z",
    "containerStatuses": [
      {
        "name": "sg-elastic-data",
        "state": {
          "running": {
            "startedAt": "2019-05-16T11:47:35Z"
          }
        },
        "lastState": {
          "terminated": {
            "exitCode": 1,
            "reason": "Error",
            "startedAt": "2019-05-16T11:42:26Z",
            "finishedAt": "2019-05-16T11:42:26Z",
            "containerID": "docker://7f2401df78ee6a58fbe273e28488df0d9cdadff2e5dc835a154408a7eddc6572"
          }
        },
        "ready": false,
        "restartCount": 563,
        "image": "bcmt-registry:5000/master/elasticsearch:build_3010",
        "imageID": "docker-pullable://bcmt-registry:5000/master/elasticsearch@sha256:52086eb788850e261c1b86ec04efb18db97ca80ab0827a796c9a7d249ddde24f",
        "containerID": "docker://a361b1c2526bcb7ae6a97070245cf86adf3a6d3208893a3df2bc3f23d04b5538"
      }
    ],
    "qosClass": "Burstable"
  }
}

1. Add lable parent-hostname to pod sg-elastic-client-3 manually， still fail $ kubectl label pod sg-elastic-client-3 -nanalytics parent-hostname=grey-manager-2 Error from server (Forbidden): pods “sg-elastic-client-3” is forbidden: unable to validate against any pod security policy: [spec.volumes[3].hostPath.pathPrefix: Invalid value: “/var/esdata/es_client”: is not allowed to be used spec.volumes[4].hostPath.pathPrefix: Invalid value: “/es_backup”: is not allowed to be used]

But psp is allowed to mount all volumes.

$ kubectl get psp privileged -o yaml
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy/v1beta1","kind":"PodSecurityPolicy","metadata":{"annotations":{"seccomp.security.alpha.kubernetes.io/allowedProfileNames":"*","seccomp.security.alpha.kubernetes.io/defaultProfileName":"docker/default"},"name":"privileged"},"spec":{"allowPrivilegeEscalation":true,"allowedCapabilities":["*"],"fsGroup":{"rule":"RunAsAny"},"hostIPC":true,"hostNetwork":true,"hostPID":true,"hostPorts":[{"max":65535,"min":0}],"privileged":true,"runAsUser":{"rule":"RunAsAny"},"seLinux":{"rule":"RunAsAny"},"supplementalGroups":{"rule":"RunAsAny"},"volumes":["*"]}}
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
  creationTimestamp: "2019-05-10T06:08:16Z"
  name: privileged
  resourceVersion: "5235622"
  selfLink: /apis/extensions/v1beta1/podsecuritypolicies/privileged
  uid: 002476a5-72ea-11e9-8d13-9418820b408c
spec:
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - '*'
  fsGroup:
    rule: RunAsAny
  hostIPC: true
  hostNetwork: true
  hostPID: true
  hostPorts:
  - max: 65535
    min: 0
  privileged: true
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - '*'

What you expected to happen: pod is in Running state after upgrade kubernetes

How to reproduce it (as minimally and precisely as possible): 1.define multile psp

$ kubectl get psp
NAME         PRIV    CAPS   SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
bcmt         false   *      RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            *
privileged   true    *      RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            *
restricted   false          RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath

1. deploy app pod on kubernetes v1.12.3
2. upgrade kubernetes from v1.12.3 to v1.13.4

Anything else we need to know?:

1. In the same statefulset, only one pod is crash, other pods 're Running, the difference is other pods 're using psp bcmt, the crash pod is using privileged, but both these two psp allow to mount all volume.

$ kubectl get psp bcmt -o yaml
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy/v1beta1","kind":"PodSecurityPolicy","metadata":{"annotations":{"seccomp.security.alpha.kubernetes.io/allowedProfileNames":"*","seccomp.security.alpha.kubernetes.io/defaultProfileName":"docker/default"},"name":"bcmt"},"spec":{"allowPrivilegeEscalation":false,"allowedCapabilities":["*"],"fsGroup":{"rule":"RunAsAny"},"hostIPC":false,"hostNetwork":false,"hostPID":false,"privileged":false,"runAsUser":{"rule":"RunAsAny"},"seLinux":{"rule":"RunAsAny"},"supplementalGroups":{"rule":"RunAsAny"},"volumes":["*"]}}
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
  creationTimestamp: "2019-05-10T06:08:16Z"
  name: bcmt
  resourceVersion: "5235632"
  selfLink: /apis/extensions/v1beta1/podsecuritypolicies/bcmt
  uid: 007bfe65-72ea-11e9-8d13-9418820b408c
spec:
  allowPrivilegeEscalation: false
  allowedCapabilities:
  - '*'
  fsGroup:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - '*'
$ kubectl get psp privileged -o yaml
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy/v1beta1","kind":"PodSecurityPolicy","metadata":{"annotations":{"seccomp.security.alpha.kubernetes.io/allowedProfileNames":"*","seccomp.security.alpha.kubernetes.io/defaultProfileName":"docker/default"},"name":"privileged"},"spec":{"allowPrivilegeEscalation":true,"allowedCapabilities":["*"],"fsGroup":{"rule":"RunAsAny"},"hostIPC":true,"hostNetwork":true,"hostPID":true,"hostPorts":[{"max":65535,"min":0}],"privileged":true,"runAsUser":{"rule":"RunAsAny"},"seLinux":{"rule":"RunAsAny"},"supplementalGroups":{"rule":"RunAsAny"},"volumes":["*"]}}
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
  creationTimestamp: "2019-05-10T06:08:16Z"
  name: privileged
  resourceVersion: "5235622"
  selfLink: /apis/extensions/v1beta1/podsecuritypolicies/privileged
  uid: 002476a5-72ea-11e9-8d13-9418820b408c
spec:
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - '*'
  fsGroup:
    rule: RunAsAny
  hostIPC: true
  hostNetwork: true
  hostPID: true
  hostPorts:
  - max: 65535
    min: 0
  privileged: true
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - '*'

1. kill the crashed pod, the new created one is Running.

Environment:

• Kubernetes version (use kubectl version): v1.13.4
• Cloud provider or hardware configuration: openstack
• OS (e.g: cat /etc/os-release): Red Hat Enterprise Linux Server release 7.5 (Maipo)
• Kernel (e.g. uname -a): Linux grey-manager-1 3.10.0-862.14.4.el7.x86_64 #1 SMP Fri Sep 21 09:07:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others: