kyverno: [BUG] Resource deletion behavior inconsistent across deny rules; fails on `foreach` rules

Software version numbers State the version numbers of applications involved in the bug.

  • Kubernetes version: 1.23.3
  • Kubernetes platform (if applicable; ex., EKS, GKE, OpenShift): K3d
  • Kyverno version: 1.6-dev-latest (1.6-dev-101-gb2724811)

Describe the bug The behavior of a DELETE request is inconsistent when comparing two deny rules.

  1. A resource validated (allowed) by a validate.deny rule is allowed to be deleted successfully with no precondition.
  2. A resource validated (allowed) by a validate.foreach.deny rule is not allowed to be deleted successfully unless there is a precondition which filters request.operation by either CREATE or UPDATE. It seems the wrong object may be being validated by the webhook.

To Reproduce Steps to reproduce the behavior:

  1. Create this policy.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-volume-types
  annotations:
    policies.kyverno.io/category: Pod Security Standards (Restricted)
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod,Volume
    policies.kyverno.io/minversion: 1.6.0
    kyverno.io/kubernetes-version: "1.22-1.23"
    kyverno.io/kyverno-version: 1.6.0
    policies.kyverno.io/description: >-
      In addition to restricting HostPath volumes, the restricted pod security profile
      limits usage of non-core volume types to those defined through PersistentVolumes.
      This policy blocks a number of different non-core volume types as named.
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: restricted-volumes
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: >-
        Any of the following volume types are restricted and may not be used: gcePersistentDisk,
        awsElasticBlockStore, gitRepo, nfs, iscsi, glusterfs, rbd, flexVolume, cinder, cephfs,
        flocker, fc, azureFile, vsphereVolume, quobyte, azureDisk, portworxVolume, scaleIO,
        storageos, and photonPersistentDisk.
      deny:
        conditions:
          all:
          - key: "{{ request.object.spec.volumes[].keys(@)[] || '' }}"
            operator: AnyIn
            value:
            - gcePersistentDisk
            - awsElasticBlockStore
            - gitRepo
            - nfs
            - iscsi
            - glusterfs
            - rbd
            - flexVolume
            - cinder
            - cephfs
            - flocker
            - fc
            - azureFile
            - vsphereVolume
            - quobyte
            - azureDisk
            - portworxVolume
            - scaleIO
            - storageos
            - photonPersistentDisk
  1. Create this good resource which will be allowed.
apiVersion: v1
kind: Pod
metadata:
  name: goodpod01
spec:
  containers:
  - name: container01
    image: dummyimagename
  1. Delete the above resource. This will also be allowed.
  2. Delete the previous ClusterPolicy.
  3. Create this ClusterPolicy below.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-capabilities-strict
  annotations:
    policies.kyverno.io/category: Pod Security Standards (Restricted)
    policies.kyverno.io/severity: medium
    policies.kyverno.io/minversion: 1.6.0
    kyverno.io/kyverno-version: 1.6.0
    kyverno.io/kubernetes-version: "1.22-1.23"
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      Adding capabilities other than `NET_BIND_SERVICE` is disallowed. In addition,
      all containers must explicitly drop `ALL` capabilities.
spec:
  validationFailureAction: enforce
  background: true
  rules:
    - name: require-drop-all
      match:
        resources:
          kinds:
            - Pod
      validate:
        message: >-
          Containers must drop `ALL` capabilities.
        foreach:
          - list: request.object.spec.[ephemeralContainers, initContainers, containers][]
            deny:
              conditions:
                all:
                - key: ALL
                  operator: AnyNotIn
                  value: "{{ element.securityContext.capabilities.drop || '' }}"
    - name: adding-capabilities-strict
      match:
        resources:
          kinds:
            - Pod
      validate:
        message: >-
          Any capabilities added other than NET_BIND_SERVICE are disallowed.
        foreach:
          - list: request.object.spec.[ephemeralContainers, initContainers, containers][]
            deny:
              conditions:
                all:
                - key: "{{ element.securityContext.capabilities.add[] || '' }}"
                  operator: AnyNotIn
                  value:
                  - NET_BIND_SERVICE
                  - ''
  1. Create this good resource. It will be allowed.
apiVersion: v1
kind: Pod
metadata:
  name: goodpod01
spec:
  containers:
  - name: container01
    image: dummyimagename
    securityContext:
      capabilities:
        drop:
        - ALL
  1. Attempt to delete the above resource and see the delete is blocked:
Error from server: error when deleting "temp.yaml": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Pod/default/goodpod01 was blocked due to the following policies

disallow-capabilities-strict:
  require-drop-all: 'validation failure: Containers must drop `ALL` capabilities.'
  1. Notice how, in both cases, the resource to be deleted was validated upon creation, but only the foreach rule blocks its deletion.

Expected behavior Delete behavior is consistent across both rules types and allowed in both cases without need for a global (policy-level) precondition.

Additional context Pertains to (but does not block) the PSS policy rewrites.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 15 (10 by maintainers)

Most upvoted comments

This was fixed in the kyverno-policies chart in PR #4046. When installing with the restricted profile, the disallow-capabilities-strict ClusterPolicy will include the precondition which fixes this behavior. The latest version of that chart is v2.4.1 which I just confirmed.

And a thank you to @treydock for that piece of work.

You can view several at kyverno.io/policies. It’s easier to filter by version from 1.5 onwards since that’s where foreach first showed up.