kyverno: [BUG] Resource deletion behavior inconsistent across deny rules; fails on `foreach` rules
Software version numbers State the version numbers of applications involved in the bug.
- Kubernetes version: 1.23.3
- Kubernetes platform (if applicable; ex., EKS, GKE, OpenShift): K3d
- Kyverno version: 1.6-dev-latest (1.6-dev-101-gb2724811)
Describe the bug
The behavior of a DELETE
request is inconsistent when comparing two deny rules.
- A resource validated (allowed) by a
validate.deny
rule is allowed to be deleted successfully with no precondition. - A resource validated (allowed) by a
validate.foreach.deny
rule is not allowed to be deleted successfully unless there is a precondition which filtersrequest.operation
by eitherCREATE
orUPDATE
. It seems the wrong object may be being validated by the webhook.
To Reproduce Steps to reproduce the behavior:
- Create this policy.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-volume-types
annotations:
policies.kyverno.io/category: Pod Security Standards (Restricted)
policies.kyverno.io/severity: medium
policies.kyverno.io/subject: Pod,Volume
policies.kyverno.io/minversion: 1.6.0
kyverno.io/kubernetes-version: "1.22-1.23"
kyverno.io/kyverno-version: 1.6.0
policies.kyverno.io/description: >-
In addition to restricting HostPath volumes, the restricted pod security profile
limits usage of non-core volume types to those defined through PersistentVolumes.
This policy blocks a number of different non-core volume types as named.
spec:
validationFailureAction: enforce
background: true
rules:
- name: restricted-volumes
match:
resources:
kinds:
- Pod
validate:
message: >-
Any of the following volume types are restricted and may not be used: gcePersistentDisk,
awsElasticBlockStore, gitRepo, nfs, iscsi, glusterfs, rbd, flexVolume, cinder, cephfs,
flocker, fc, azureFile, vsphereVolume, quobyte, azureDisk, portworxVolume, scaleIO,
storageos, and photonPersistentDisk.
deny:
conditions:
all:
- key: "{{ request.object.spec.volumes[].keys(@)[] || '' }}"
operator: AnyIn
value:
- gcePersistentDisk
- awsElasticBlockStore
- gitRepo
- nfs
- iscsi
- glusterfs
- rbd
- flexVolume
- cinder
- cephfs
- flocker
- fc
- azureFile
- vsphereVolume
- quobyte
- azureDisk
- portworxVolume
- scaleIO
- storageos
- photonPersistentDisk
- Create this good resource which will be allowed.
apiVersion: v1
kind: Pod
metadata:
name: goodpod01
spec:
containers:
- name: container01
image: dummyimagename
- Delete the above resource. This will also be allowed.
- Delete the previous ClusterPolicy.
- Create this ClusterPolicy below.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-capabilities-strict
annotations:
policies.kyverno.io/category: Pod Security Standards (Restricted)
policies.kyverno.io/severity: medium
policies.kyverno.io/minversion: 1.6.0
kyverno.io/kyverno-version: 1.6.0
kyverno.io/kubernetes-version: "1.22-1.23"
policies.kyverno.io/subject: Pod
policies.kyverno.io/description: >-
Adding capabilities other than `NET_BIND_SERVICE` is disallowed. In addition,
all containers must explicitly drop `ALL` capabilities.
spec:
validationFailureAction: enforce
background: true
rules:
- name: require-drop-all
match:
resources:
kinds:
- Pod
validate:
message: >-
Containers must drop `ALL` capabilities.
foreach:
- list: request.object.spec.[ephemeralContainers, initContainers, containers][]
deny:
conditions:
all:
- key: ALL
operator: AnyNotIn
value: "{{ element.securityContext.capabilities.drop || '' }}"
- name: adding-capabilities-strict
match:
resources:
kinds:
- Pod
validate:
message: >-
Any capabilities added other than NET_BIND_SERVICE are disallowed.
foreach:
- list: request.object.spec.[ephemeralContainers, initContainers, containers][]
deny:
conditions:
all:
- key: "{{ element.securityContext.capabilities.add[] || '' }}"
operator: AnyNotIn
value:
- NET_BIND_SERVICE
- ''
- Create this good resource. It will be allowed.
apiVersion: v1
kind: Pod
metadata:
name: goodpod01
spec:
containers:
- name: container01
image: dummyimagename
securityContext:
capabilities:
drop:
- ALL
- Attempt to delete the above resource and see the delete is blocked:
Error from server: error when deleting "temp.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:
resource Pod/default/goodpod01 was blocked due to the following policies
disallow-capabilities-strict:
require-drop-all: 'validation failure: Containers must drop `ALL` capabilities.'
- Notice how, in both cases, the resource to be deleted was validated upon creation, but only the
foreach
rule blocks its deletion.
Expected behavior Delete behavior is consistent across both rules types and allowed in both cases without need for a global (policy-level) precondition.
Additional context Pertains to (but does not block) the PSS policy rewrites.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 15 (10 by maintainers)
This was fixed in the
kyverno-policies
chart in PR #4046. When installing with therestricted
profile, thedisallow-capabilities-strict
ClusterPolicy will include the precondition which fixes this behavior. The latest version of that chart is v2.4.1 which I just confirmed.And a thank you to @treydock for that piece of work.
You can view several at kyverno.io/policies. It’s easier to filter by version from 1.5 onwards since that’s where
foreach
first showed up.