kyverno: [BUG] Kyverno CLI `test` command and `-f` flag

Software version numbers State the version numbers of applications involved in the bug.

  • Kubernetes version: 1.21
  • Kubernetes platform (if applicable; ex., EKS, GKE, OpenShift): K3d
  • Kyverno version: 1.4.2

Describe the bug When passing an argument to the kyverno test -f flag, the name of the test file, regardless if it’s specified as a relative path or full path, is not accepted and Kyverno CLI prints Error: a directory is required.

To Reproduce Steps to reproduce the behavior:

  1. Have a directory containing a test file, policy, and resource as outlined in #2303
  2. cd to the directory having these files.
  3. Run kyverno test -f test.yaml
  4. See error.
  5. Format the file path with a fully-qualified path like kyverno test -f /foo/bar/test.yaml.
  6. See error.

Expected behavior The kyverno test -f command accepts relative and fully-qualified paths to a test file of the specified name. The test manifest’s references to files should be assumed to be relative paths unless otherwise specified.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 19 (18 by maintainers)

Most upvoted comments

@NoSkillGirl I’ve edited the above output. See what you think:

The `test` command provides a facility to test resources against policies by comparing expected results, declared ahead of time
in a `test.yaml` file, to actual results reported by Kyverno. Users provide the path to the folder containing a `test.yaml` file where the
location could be on a local filesystem or a remote git repository.

Usage:

    kyverno test <path_to_folder_Containing_test.yamls> [flags]
    kyverno test <path_to_gitRepository> [flags]

Examples:

    kyverno test https://github.com/kyverno/policies/main
    <snip>

    Executing disallow-cri-sock-mount...
    applying 1 policy to 1 resource...
    │───│────────────────────────────────│────────────────────────────────│────────────────────────────│────────│
    │ # │ POLICY                         │ RULE                           │ RESOURCE                   │ RESULT │
    │───│────────────────────────────────│────────────────────────────────│────────────────────────────│────────│
    │ 1 │ disallow-container-sock-mounts │ validate-docker-sock-mount     │ pod-with-docker-sock-mount │ Pass   │
    │ 2 │ disallow-container-sock-mounts │ validate-containerd-sock-mount │ pod-with-docker-sock-mount │ Pass   │
    │ 3 │ disallow-container-sock-mounts │ validate-crio-sock-mount       │ pod-with-docker-sock-mount │ Pass   │
    │───│────────────────────────────────│────────────────────────────────│────────────────────────────│────────│
    <snip>


Test file structure:

The test.yaml has four parts:
    "policies"   --> List of policies which are applied.
    "resources"  --> List of resources on which the policies are applied.
    "variables"  --> Variable file path (optional).
    "results"    --> List of results expected after applying the policies on the resources.

Test file format:

For validate policies

- name: test-1
  policies:
  - <path>
  - <path>
  resources:
  - <path>
  - <path>
  results:
  - policy: <name>
    rule: <name>
    resource: <name>
    namespace: <name> (OPTIONAL)
    kind: <name> 
    result: <pass|fail|skip>


For mutate policies

Policy (Namespaced)

- name: test-1
  policies:
  - <path>
  - <path>
  resources:
  - <path>
  - <path>
  results:
  - policy: <policy_namespace>/<policy_name>
    rule: <name>
    resource: <name>
    namespace: <name> (OPTIONAL)
        kind: <name> 
    patchedResource: <path>
    result: <pass|fail|skip>

ClusterPolicy (Cluster-wide)

- name: test-1
  policies:
  - <path>
  - <path>
  resources:
  - <path>
  - <path>
  results:
  - policy: <name>
    rule: <name>
    resource: <name>
    namespace: <name> (OPTIONAL)
    kind: <name> 
    patchedResource: <path>
    result: <pass|fail|skip>

Result descriptions:

pass  --> The patched resource generated by Kyverno equals the patched resource provided by the user.
fail  --> The patched resource generated by Kyverno is not equal to the patched resource provided by the user. 
skip  --> The rule is not applied.

For more information visit https://kyverno.io/docs/kyverno-cli/#test


Flags:

    -f, --file-name string   test filename (default "test.yaml")
    -h, --help               help for test

Global Flags:

    --add_dir_header                           If true, adds the file directory to the header of the log messages
    --azure-container-registry-config string   Path to the file containing Azure container registry configuration information.
    --log_file string                          If non-empty, use this log file
    --log_file_max_size uint                   Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
    --one_output                               If true, only write logs to their native severity level (vs also writing to each lower severity level)
    --skip_headers                             If true, avoid header prefixes in the log messages
    --skip_log_headers                         If true, avoid headers when opening 

This is a great improvement, @NoSkillGirl, but there is some polishing needed both in language and examples. Let me respond a bit later with some suggestions.

The file is found. It’s there.