kyverno: [BUG] Resource deletion behavior inconsistent across deny rules; fails on `foreach` rules
Software version numbers State the version numbers of applications involved in the bug.
- Kubernetes version: 1.23.3
- Kubernetes platform (if applicable; ex., EKS, GKE, OpenShift): K3d
- Kyverno version: 1.6-dev-latest (1.6-dev-101-gb2724811)
Describe the bug
The behavior of a DELETE request is inconsistent when comparing two deny rules.
- A resource validated (allowed) by a
validate.denyrule is allowed to be deleted successfully with no precondition. - A resource validated (allowed) by a
validate.foreach.denyrule is not allowed to be deleted successfully unless there is a precondition which filtersrequest.operationby eitherCREATEorUPDATE. It seems the wrong object may be being validated by the webhook.
To Reproduce Steps to reproduce the behavior:
- Create this policy.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-volume-types
annotations:
policies.kyverno.io/category: Pod Security Standards (Restricted)
policies.kyverno.io/severity: medium
policies.kyverno.io/subject: Pod,Volume
policies.kyverno.io/minversion: 1.6.0
kyverno.io/kubernetes-version: "1.22-1.23"
kyverno.io/kyverno-version: 1.6.0
policies.kyverno.io/description: >-
In addition to restricting HostPath volumes, the restricted pod security profile
limits usage of non-core volume types to those defined through PersistentVolumes.
This policy blocks a number of different non-core volume types as named.
spec:
validationFailureAction: enforce
background: true
rules:
- name: restricted-volumes
match:
resources:
kinds:
- Pod
validate:
message: >-
Any of the following volume types are restricted and may not be used: gcePersistentDisk,
awsElasticBlockStore, gitRepo, nfs, iscsi, glusterfs, rbd, flexVolume, cinder, cephfs,
flocker, fc, azureFile, vsphereVolume, quobyte, azureDisk, portworxVolume, scaleIO,
storageos, and photonPersistentDisk.
deny:
conditions:
all:
- key: "{{ request.object.spec.volumes[].keys(@)[] || '' }}"
operator: AnyIn
value:
- gcePersistentDisk
- awsElasticBlockStore
- gitRepo
- nfs
- iscsi
- glusterfs
- rbd
- flexVolume
- cinder
- cephfs
- flocker
- fc
- azureFile
- vsphereVolume
- quobyte
- azureDisk
- portworxVolume
- scaleIO
- storageos
- photonPersistentDisk
- Create this good resource which will be allowed.
apiVersion: v1
kind: Pod
metadata:
name: goodpod01
spec:
containers:
- name: container01
image: dummyimagename
- Delete the above resource. This will also be allowed.
- Delete the previous ClusterPolicy.
- Create this ClusterPolicy below.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-capabilities-strict
annotations:
policies.kyverno.io/category: Pod Security Standards (Restricted)
policies.kyverno.io/severity: medium
policies.kyverno.io/minversion: 1.6.0
kyverno.io/kyverno-version: 1.6.0
kyverno.io/kubernetes-version: "1.22-1.23"
policies.kyverno.io/subject: Pod
policies.kyverno.io/description: >-
Adding capabilities other than `NET_BIND_SERVICE` is disallowed. In addition,
all containers must explicitly drop `ALL` capabilities.
spec:
validationFailureAction: enforce
background: true
rules:
- name: require-drop-all
match:
resources:
kinds:
- Pod
validate:
message: >-
Containers must drop `ALL` capabilities.
foreach:
- list: request.object.spec.[ephemeralContainers, initContainers, containers][]
deny:
conditions:
all:
- key: ALL
operator: AnyNotIn
value: "{{ element.securityContext.capabilities.drop || '' }}"
- name: adding-capabilities-strict
match:
resources:
kinds:
- Pod
validate:
message: >-
Any capabilities added other than NET_BIND_SERVICE are disallowed.
foreach:
- list: request.object.spec.[ephemeralContainers, initContainers, containers][]
deny:
conditions:
all:
- key: "{{ element.securityContext.capabilities.add[] || '' }}"
operator: AnyNotIn
value:
- NET_BIND_SERVICE
- ''
- Create this good resource. It will be allowed.
apiVersion: v1
kind: Pod
metadata:
name: goodpod01
spec:
containers:
- name: container01
image: dummyimagename
securityContext:
capabilities:
drop:
- ALL
- Attempt to delete the above resource and see the delete is blocked:
Error from server: error when deleting "temp.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:
resource Pod/default/goodpod01 was blocked due to the following policies
disallow-capabilities-strict:
require-drop-all: 'validation failure: Containers must drop `ALL` capabilities.'
- Notice how, in both cases, the resource to be deleted was validated upon creation, but only the
foreachrule blocks its deletion.
Expected behavior Delete behavior is consistent across both rules types and allowed in both cases without need for a global (policy-level) precondition.
Additional context Pertains to (but does not block) the PSS policy rewrites.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 15 (10 by maintainers)
This was fixed in the
kyverno-policieschart in PR #4046. When installing with therestrictedprofile, thedisallow-capabilities-strictClusterPolicy will include the precondition which fixes this behavior. The latest version of that chart is v2.4.1 which I just confirmed.And a thank you to @treydock for that piece of work.
You can view several at kyverno.io/policies. It’s easier to filter by version from 1.5 onwards since that’s where
foreachfirst showed up.