kyverno: [Bug] Kyverno Policy passed in enforce mode but not in audit mode
Kyverno Version
1.7.2
Kubernetes Version
1.23.x
Kubernetes Platform
EKS
Kyverno Rule Type
Validate
Description
Hi,
we noticed some problems with the require-drop-all policy if the validationFailureAction is set on “audit”. The capabilities are set correct, but the validation failed. If we switch the validationFailureAction from “audit” to “enforce” it passed. Hope you are able to identify the problem.
Steps to reproduce
We tested the following on EKS 1.22 and 1.23.
- We Apply the Policy and a minute later the STS. The first POD looks good. Don’t see a failed event for the require-drop-all Policy.
require-drop-all Policy
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
annotations:
pod-policies.kyverno.io/autogen-controllers: none
policies.kyverno.io/category: Security
policies.kyverno.io/description: 'Capabilities permit privileged actions without
giving full root access. All capabilities should be dropped from a Pod, with
only those required added back. This policy ensures that all containers explicitly
specify the `drop: ["ALL"]` ability. '
policies.kyverno.io/minversion: 1.6.0
policies.kyverno.io/severity: medium
policies.kyverno.io/subject: Pod
policies.kyverno.io/title: Drop All Capabilities
name: require-drop-all
spec:
background: true
rules:
- match:
any:
- resources:
kinds:
- Pod
namespaceSelector:
matchExpressions:
- key: managed
operator: In
values:
- "true"
- resources:
kinds:
- Pod
namespaces:
- default
name: require-drop-all
preconditions:
all:
- key: '{{ request.operation }}'
operator: NotEquals
value: DELETE
validate:
foreach:
- deny:
conditions:
all:
- key: ALL
operator: AnyNotIn
value: '{{ element.securityContext.capabilities.drop || '''' }}'
list: request.object.spec.[ephemeralContainers, initContainers, containers][]
message: Containers must drop `ALL` capabilities.
validationFailureAction: audit
Jenkins STS
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
bu: a-it
run: jenkins-statefulset
service: jenkins
name: ea-test-ds2-jenkins-statefulset
namespace: default
spec:
podManagementPolicy: OrderedReady
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
run: ea-test-jenkins-pod
serviceName: ea-test-jenkins-svc
template:
metadata:
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
creationTimestamp: null
labels:
backup: nfs
bu: a-it
run: ea-test-jenkins-pod
service: jenkins
spec:
containers:
- env:
- name: CONTAINER_HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: TZ
value: Europe/Berlin
image: jenkins/jenkins:2.346.1
imagePullPolicy: IfNotPresent
name: jenkins
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
memory: 2Gi
requests:
memory: 2Gi
securityContext:
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/jenkins_home
name: jenkins-home
- args:
- java
- -jar
- /usr/share/jmx_exporter/jmx_prometheus_httpserver-0.12.0-jar-with-dependencies.jar
- "9070"
- /var/ea/jmx.yml
env:
- name: SERVICE_PORT
value: "9070"
- name: CONFIG_YML
value: /var/ea/jmx.yml
image: koarch/jmx-exporter:0.12.0
imagePullPolicy: IfNotPresent
name: jmx
ports:
- containerPort: 9070
protocol: TCP
resources: {}
securityContext:
capabilities:
drop:
- ALL
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
nodeSelector:
kubernetes.io/arch: arm64
node.kubernetes.io/instance-type: t4g.medium
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: arm64
operator: Equal
value: "true"
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
labels:
bu: ea
service: jenkins
type: amazonEBS
name: jenkins-home
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
volumeMode: Filesystem
- We delete the POD to start a new one.
% kubectl delete pod ea-test-ds2-jenkins-statefulset-0
pod "ea-test-ds2-jenkins-statefulset-0" deleted
- Run a Describe on the Pod, Policy require-drop-all failed
% kubectl describe pod ea-test-ds2-jenkins-statefulset-0
Name: ea-test-ds2-jenkins-statefulset-0
Namespace: default
Priority: 0
Start Time: Fri, 19 Aug 2022 16:52:42 +0200
....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning PolicyViolation 27s kyverno-admission policy require-drop-all/require-drop-all fail: validation failure: Containers must drop `ALL` capabilities.
Warning PolicyViolation 27s kyverno-admission policy require-run-as-nonroot/run-as-non-root fail: validation error: Running as root is not allowed. Either the field spec.securityContext.runAsNonRoot must be set to `true`, or the fields spec.containers[*].securityContext.runAsNonRoot, spec.initContainers[*].securityContext.runAsNonRoot, and spec.ephemeralContainers[*].securityContext.runAsNonRoot must be set to `true`. . Rule run-as-non-root[0] failed at path /spec/securityContext/runAsNonRoot/. Rule run-as-non-root[1] failed at path /spec/containers/1/securityContext/runAsNonRoot/.
Normal Pulled 18s kubelet Container image "jenkins/jenkins:2.346.1" already present on machine
Normal Created 18s kubelet Created container jenkins
Normal Started 17s kubelet Started container jenkins
- Switch Action to enforce
% kubectl patch clusterpolicy require-drop-all --type='json' -p='[{"op": "replace", "path": "/spec/validationFailureAction", "value":"enforce"}]'
clusterpolicy.kyverno.io/require-drop-all patched
- We delete the POD again to start a new one.
% kubectl delete pod ea-test-ds2-jenkins-statefulset-0
pod "ea-test-ds2-jenkins-statefulset-0" deleted
- Run a Describe on the Pod, Policy require-drop-all passed PolicyApplied passed.
% kubectl describe pod ea-test-ds2-jenkins-statefulset-0
Name: ea-test-ds2-jenkins-statefulset-0
Namespace: default
Start Time: Fri, 19 Aug 2022 16:54:38 +0200
....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning PolicyViolation 12s kyverno-admission policy require-run-as-nonroot/run-as-non-root fail: validation error: Running as root is not allowed. Either the field spec.securityContext.runAsNonRoot must be set to `true`, or the fields spec.containers[*].securityContext.runAsNonRoot, spec.initContainers[*].securityContext.runAsNonRoot, and spec.ephemeralContainers[*].securityContext.runAsNonRoot must be set to `true`. . Rule run-as-non-root[0] failed at path /spec/securityContext/runAsNonRoot/. Rule run-as-non-root[1] failed at path /spec/containers/1/securityContext/runAsNonRoot/.
Warning FailedMount 11s kubelet MountVolume.SetUp failed for volume "kube-api-access-54fbl" : failed to sync configmap cache: timed out waiting for the condition
Normal Pulled 7s kubelet Container image "jenkins/jenkins:2.346.1" already present on machine
Normal Created 7s kubelet Created container jenkins
Normal Started 7s kubelet Started container jenkins
- We switched Action back to audit
% kubectl patch clusterpolicy require-drop-all --type='json' -p='[{"op": "replace", "path": "/spec/validationFailureAction", "value":"audit"}]'
clusterpolicy.kyverno.io/require-drop-all patched
- We delete the POD again to start a new one.
% kubectl delete pod ea-test-ds2-jenkins-statefulset-0
pod "ea-test-ds2-jenkins-statefulset-0" deleted
- Run a Describe on the Pod, Policy require-drop-all failed
% kubectl describe pod ea-test-ds2-jenkins-statefulset-0
Name: ea-test-ds2-jenkins-statefulset-0
Namespace: default
Start Time: Fri, 19 Aug 2022 16:55:56 +0200
....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning PolicyViolation 7s (x2 over 7s) kyverno-admission policy require-drop-all/require-drop-all fail: validation failure: Containers must drop `ALL` capabilities.
Warning PolicyViolation 7s kyverno-admission policy require-run-as-nonroot/run-as-non-root fail: validation error: Running as root is not allowed. Either the field spec.securityContext.runAsNonRoot must be set to `true`, or the fields spec.containers[*].securityContext.runAsNonRoot, spec.initContainers[*].securityContext.runAsNonRoot, and spec.ephemeralContainers[*].securityContext.runAsNonRoot must be set to `true`. . Rule run-as-non-root[0] failed at path /spec/securityContext/runAsNonRoot/. Rule run-as-non-root[1] failed at path /spec/containers/1/securityContext/runAsNonRoot/.
- Output of the require-drop-all Policy. PolicyViolation failed, but PolicyApplied passed.
% kubectl describe clusterpolicy require-drop-all
Name: require-drop-all
Namespace:
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: kyverno-cmo-policies
meta.helm.sh/release-namespace: kyverno
pod-policies.kyverno.io/autogen-controllers: none
policies.kyverno.io/category: Security
policies.kyverno.io/description:
Capabilities permit privileged actions without giving full root access. All capabilities should be dropped from a Pod, with only those req...
policies.kyverno.io/minversion: 1.6.0
policies.kyverno.io/severity: medium
policies.kyverno.io/subject: Pod
policies.kyverno.io/title: Drop All Capabilities
API Version: kyverno.io/v1
Kind: ClusterPolicy
....
Status:
Conditions:
Last Transition Time: 2022-08-16T14:40:00Z
Message:
Reason: Succeeded
Status: True
Type: Ready
Ready: true
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal PolicyApplied 15m (x24 over 140m) kyverno-admission Pod default/ea-test-ds2-jenkins-statefulset-0: pass
Normal PolicyApplied 10m (x19 over 140m) kyverno-admission Pod default/ea-test-ds2-jenkins-statefulset-0: pass
Warning PolicyViolation 10m (x12 over 99m) kyverno-admission Pod default/ea-test-ds2-jenkins-statefulset-0: [require-drop-all] fail
Warning PolicyViolation 4m37s (x16 over 112m) kyverno-admission Pod default/ea-test-ds2-jenkins-statefulset-0: [require-drop-all] fail
Normal PolicyApplied 105s (x13 over 141m) kyverno-admission Pod default/ea-test-ds2-jenkins-statefulset-0: pass
Expected behavior
Same behavior for audit and enforce mode.
Screenshots
No response
Kyverno logs
No response
Slack discussion
No response
Troubleshooting
- I have read and followed the documentation AND the troubleshooting guide.
- I have searched other issues in this repository and mine is not recorded.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 20 (10 by maintainers)
Hi, thanks for the update. I will test it today.