kyverno: [Bug] [CLI] 1.11.0 CLI breaks test for rules with exclusions

Kyverno CLI Version

1.11.0-rc.5

Description

Hello,

I have a CI pipeline on my gitops repository that runs all kyverno tests created in the repo. Since the push of version 1.10.5 [edit by @chipzoller: this was actually 1.11.0-rc.5] my CI now installs it as krew does not allow plugin version pinning and all my pipelines fail.

Steps to reproduce

  1. create a policy with a rule that uses exclusions require-requests-limits.yaml:
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-requests-limits
  annotations:
    policies.kyverno.io/title: Require Limits and Requests
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      As application workloads share cluster resources, it is important to limit resources
      requested and consumed by each Pod. It is recommended to require resource requests and
      limits per Pod, especially for memory and CPU. If a Namespace level request or limit is specified,
      defaults will automatically be applied to each Pod based on the LimitRange configuration.
      This policy validates that all containers have something specified for memory and CPU
      requests and memory limits.
spec:
  background: true
  validationFailureAction: enforce
  rules:
    - name: validate-resources
      match:
        any:
          - resources:
              kinds:
                - Pod
      exclude:
        any:
          - resources:
              namespaces:
                - ceph-csi
                - metallb-system
          - resources:
              kinds:
                - Pod
              selector:
                matchLabels:
                  require-requests-limits.kyverno.io/exclude: "true"
      validate:
        message: "CPU and memory resource requests and limits are required."
        pattern:                                                                                                                                                                   
          spec:                                                                                                                                                                    
            containers:                                                                                                                                                            
              - resources:                                                                                                                                                         
                  requests:                                                                                                                                                        
                    memory: "?*"                                                                                                                                                   
                    cpu: "?*"                                                                                                                                                      
                  limits:                                                                                                                                                          
                    memory: "?*"                                                                                                                                                   
                    cpu: "?*"
  1. Create some tests kyverno-test.yaml:
---
name: require-requests-limits
policies:
  - require-requests-limits.yaml
resources:
  - ./resources.yaml
results:
  - policy: require-requests-limits
    rule: validate-resources
    resources:
      - require-requests-limits-skip
    kind: Pod
    result: skip

  - policy: require-requests-limits
    rule: validate-resources
    resources:
      - require-requests-limits-failed
    kind: Pod
    namespace: ceph-csi
    result: skip

  - policy: require-requests-limits
    rule: validate-resources
    resources:
      - require-requests-limits-failed
    kind: Pod
    namespace: metallb-system
    result: skip

  - policy: require-requests-limits
    rule: validate-resources
    resources:
      - require-requests-limits-failed
      - require-requests-limits-only-limits-failed
      - require-requests-limits-only-requests-failed
    kind: Pod
    result: fail

  - policy: require-requests-limits
    rule: validate-resources
    resources:
      - require-requests-limits-pass
    kind: Pod
    result: pass

resources.yaml:

  ---
  # Skip check requests/limits using label
  apiVersion: v1
  kind: Pod
  metadata:
    labels:
      require-requests-limits.kyverno.io/exclude: "true"
    name: require-requests-limits-skip
    namespace: test
  spec:
    containers:
      - name: test
        image: nginx

  ---
  # Fail check requests/limits
  apiVersion: v1
  kind: Pod
  metadata:
    labels:
    name: require-requests-limits-failed
    namespace: test
  spec:
    containers:
      - name: test
        image: nginx

  ---
  # Fail check requests/limits
  apiVersion: v1
  kind: Pod
  metadata:
    labels:
    name: require-requests-limits-only-limits-failed
    namespace: test
  spec:
    containers:
      - name: test
        image: nginx
        resources:
          limits:
            memory: "256Mi"
            cpu: "1000m"

  ---
  # Fail check requests/limits
  apiVersion: v1
  kind: Pod
  metadata:
    labels:
    name: require-requests-limits-only-requests-failed
    namespace: test
  spec:
    containers:
      - name: test
        image: nginx
        resources:
          requests:
            memory: "128Mi"
            cpu: "250m"

  ---
  # Pass check requests/limits
  apiVersion: v1
  kind: Pod
  metadata:
    name: require-requests-limits-pass
    namespace: test
  spec:
    containers:
      - name: test
        image: nginx
        resources:
          requests:
            memory: "128Mi"
            cpu: "250m"
          limits:
            memory: "256Mi"
            cpu: "1000m"
  1. Run the tests with 1.10.5 CLI
kyverno test require-requests-limits
Loading test require-requests-limits ( require-requests-limits/kyverno-test.yaml ) ...
  Loading values/variables ...
  Loading policies ...
  Loading resources ...
  Applying 1 policy to 5 resources ...
  Checking results ...

│────│─────────────────────────│────────────────────│───────────────────────────────────────────────────│────────│─────────────────────│
│ ID │ POLICY                  │ RULE               │ RESOURCE                                          │ RESULT │ REASON              │
│────│─────────────────────────│────────────────────│───────────────────────────────────────────────────│────────│─────────────────────│
│  1 │ require-requests-limits │ validate-resources │ Pod/require-requests-limits-skip                  │ Fail   │ Not found           │
│  2 │ require-requests-limits │ validate-resources │ ceph-csi/Pod/require-requests-limits-failed       │ Fail   │ Want skip, got fail │
│  3 │ require-requests-limits │ validate-resources │ metallb-system/Pod/require-requests-limits-failed │ Fail   │ Want skip, got fail │
│  4 │ require-requests-limits │ validate-resources │ Pod/require-requests-limits-failed                │ Pass   │ Ok                  │
│  5 │ require-requests-limits │ validate-resources │ Pod/require-requests-limits-only-limits-failed    │ Pass   │ Ok                  │
│  6 │ require-requests-limits │ validate-resources │ Pod/require-requests-limits-only-requests-failed  │ Pass   │ Ok                  │
│  7 │ require-requests-limits │ validate-resources │ Pod/require-requests-limits-pass                  │ Pass   │ Ok                  │
│────│─────────────────────────│────────────────────│───────────────────────────────────────────────────│────────│─────────────────────│


Test Summary: 4 tests passed and 3 tests failed

Aggregated Failed Test Cases : 
│────│─────────────────────────│────────────────────│───────────────────────────────────────────────────│────────│─────────────────────│
│ ID │ POLICY                  │ RULE               │ RESOURCE                                          │ RESULT │ REASON              │
│────│─────────────────────────│────────────────────│───────────────────────────────────────────────────│────────│─────────────────────│
│  1 │ require-requests-limits │ validate-resources │ Pod/require-requests-limits-skip                  │ Fail   │ Not found           │
│  2 │ require-requests-limits │ validate-resources │ ceph-csi/Pod/require-requests-limits-failed       │ Fail   │ Want skip, got fail │
│  3 │ require-requests-limits │ validate-resources │ metallb-system/Pod/require-requests-limits-failed │ Fail   │ Want skip, got fail │
│────│─────────────────────────│────────────────────│───────────────────────────────────────────────────│────────│─────────────────────│
Error: 3 tests failed

Expected behavior

The tests results should be the same as with 1.10.3 CLI

Executing require-requests-limits...
applying 1 policy to 5 resources... 

│───│─────────────────────────│────────────────────│───────────────────────────────────────────────────────│────────│
│ # │ POLICY                  │ RULE               │ RESOURCE                                              │ RESULT │
│───│─────────────────────────│────────────────────│───────────────────────────────────────────────────────│────────│
│ 1 │ require-requests-limits │ validate-resources │ test/Pod/require-requests-limits-skip                 │ Pass   │
│ 2 │ require-requests-limits │ validate-resources │ ceph-csi/Pod/require-requests-limits-failed           │ Pass   │
│ 3 │ require-requests-limits │ validate-resources │ metallb-system/Pod/require-requests-limits-failed     │ Pass   │
│ 4 │ require-requests-limits │ validate-resources │ test/Pod/require-requests-limits-failed               │ Pass   │
│ 5 │ require-requests-limits │ validate-resources │ test/Pod/require-requests-limits-only-limits-failed   │ Pass   │
│ 6 │ require-requests-limits │ validate-resources │ test/Pod/require-requests-limits-only-requests-failed │ Pass   │
│ 7 │ require-requests-limits │ validate-resources │ test/Pod/require-requests-limits-pass                 │ Pass   │
│───│─────────────────────────│────────────────────│───────────────────────────────────────────────────────│────────│

Test Summary: 7 tests passed and 0 tests failed

Screenshots

No response

Kyverno logs

No response

Slack discussion

No response

Troubleshooting

  • I have read and followed the troubleshooting guide.
  • I have searched other issues in this repository and mine is not recorded.

About this issue

  • Original URL
  • State: closed
  • Created 8 months ago
  • Reactions: 3
  • Comments: 20 (9 by maintainers)

Most upvoted comments

Reopening as newly released 1.11.0 has the same problem and no breaking change is listed on the release notes.

+5
Da-Juan on Nov 14, 2023

@eddycharly Could we have a feedback on this issue please? It is annoying to be unable to test our rules in CI anymore.

+4
Da-Juan on Nov 22, 2023

We’ve discussed this internally and we decided on the following:

  1. Add a new Rule Response called NoMatch for excluded resources.
  2. This will not impact reports as it will not be reported.
  3. For the CLI we will report it as skip.
+1
KhaledEmaraDev on Dec 26, 2023

This is still applicable to v1.11.1

I was just trying to update to v1.11.1 but 10% of my unit tests now fail.

+1
joebowbeer on Dec 5, 2023

This is still applicable to v1.11.1

+1
aslafy-z on Dec 1, 2023
Read more comments on GitHub
← kyverno: [Bug] Cleanup based on policy can be broken
kyverno: [Bug] [CLI] Exclude statements are not evaluated correctly during testing →