trivy-operator: No warnings or failed cis benchmarks
What steps did you take and what happened: I get different results for CIS benchmarks when I use kube-bench and the trivy operator.
When I run the built-in scanner of the trivy-operator for cis benchmarks I don’t get failed checks. All 116 checks are successful, although I know that one or the other should at least throw a warning.
Prometheus output via Servicemonitor
trivy_cluster_compliance{container="trivy-operator", description="CIS Kubernetes Benchmarks", endpoint="metrics", job="trivy-operator", service="trivy-operator", status="Pass", title="CIS Kubernetes Benchmarks v1.23"} 116
Also in the clustercompliance resource I can see that no checks fail:
kubectl clustercompliancereports cis -o yaml
When I run the kube-bench job in the same cluster and context, I get the expected warnings and errors for the cis benchmarks.
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl kube-bench-<id> -f
What did you expect to happen: I expect to get the same warnings and errors for the CIS benchmarks. Regardless of which scanner I use for this.
HELM Values:
resources:
limits:
cpu: 150m
memory: 100M
trivy:
ignoreUnfixed: true
mode: ClientServer
operator:
configAuditScannerEnabled: false
vulnerabilityScannerReportTTL: "86400s"
vulnerabilityScannerScanOnlyCurrentRevisions: true
rbacAssessmentScannerEnabled: false
infraAssessmentScannerEnabled: false
exposedSecretScannerEnabled: false
compliance:
reportType: all
# for test purposes currently set every minute
cron: "* * * * *"
serviceMonitor:
enabled: true
Environment:
- Trivy-Operator (HELM Cart version):
0.14.1 - Trivy-Operator (Image version):
ghcr.io/aquasecurity/trivy-operator:0.14.1 - Kubernetes version:
v1.23.7 - OS: Linux
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 18
@tibuntu thanks for your update, I’ll make sure docs are updated accordingly
That was a good idea it seems 😃
Now several
node-collector-*pods showed up, e.g.:node-collector-8b6cf59b6-zx46zI see lots of
infraAssesmentReports:same for clusterinfraassessmentreports:
And best thing is, the CIS scan has proper results:
I am going to check if there is any difference on the trivy-operator deployment / configmaps / secrets that I can find that explains the different behaviour.
@yanehi can this issue be closed ?
Ok so here is the final result.
I kept deleting the Helm-Release via ArgoCD, also always deleted the CRDs, and then re-deployed with several, adapted values.
Got it working now with the following values:
Setting
configAuditScannerEnabled: truewas the key factor here.I am unsure if the current documentation reflects that requirement for proper clustercompliancereports?
Anyways, big thanks for your help and fast replies!
you should have something, it explain why compliance report do no show infra data
btw: are you running on managed cluster (cloud) or on prem?
I suggest to do another simple test , perform the following , lets clean up env and start fresh
@chen-keinan
This is the summary of the kube-bench job:
One of the failing checks is:
The same check passes in the report generated by trivy-operator:
this is the CIS scan summary of the trivy-operator:
If I am not mistaken we had
infraAssessmentScannerEnabledset totrueat some point. But it caused no difference.Edit:
I checked the environment variables set in the trivy-operator deployment.
IIRC I saw another issue in this repo where someone stated that setting the helm value to false doesn’t have any effect on the env vars being set on the deployment.