opa: OPA does not load custom policies

Hello,

Step 1: I have installed OPA to my Kubernetes cluster(v1.15.5) using opa helm chart(https://github.com/helm/charts/tree/master/stable/opa). I have overridden values.YAML file as below:

values.yaml:

# Default values for opa.
# -----------------------
#
# The 'opa' key embeds an OPA configuration file. See
# https://www.openpolicyagent.org/docs/configuration.html for more details.
# Default value is no default config. For custom config, the opa key
# needs to include the opa config yaml, eg:
opa:
  services:
    controller:
      url: 'https://www.openpolicyagent.org'
  bundles:
    quickstart:
      service: controller
      resource: /bundles/helm-kubernetes-quickstart
  default_decision: /helm_kubernetes_quickstart/main

# Setup the webhook using cert-manager
certManager:
  enabled: false

# Expose the prometheus scraping endpoint
prometheus:
  enabled: false

## ServiceMonitor consumed by prometheus-operator
serviceMonitor:
  ## If the operator is installed in your cluster, set to true to create a Service Monitor Entry
  enabled: false
  interval: "15s"
  ## Namespace in which the service monitor is created
  # namespace: monitoring
  # Added to the ServiceMonitor object so that prometheus-operator is able to discover it
  ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec
  additionalLabels: {}

# Annotations in the deployment template
annotations:
  {}

# Bootstrap policies to load upon startup
# Define policies in the form of:
# <policyName> : |-
#   <regoBody>
# For example, to mask the entire input body in the decision logs:
# bootstrapPolicies:
#   log: |-
#     package system.log
#     mask["/input"]
bootstrapPolicies: {}

# To enforce mutating policies, change to MutatingWebhookConfiguration.
admissionControllerKind: ValidatingWebhookConfiguration

# To _fail closed_ on failures, change to Fail. During initial testing, we
# recommend leaving the failure policy as Ignore.
admissionControllerFailurePolicy: Ignore

# Adds a namespace selector to the admission controller webhook
admissionControllerNamespaceSelector:
  matchExpressions:
    - {key: openpolicyagent.org/webhook, operator: NotIn, values: [ignore]}

# SideEffectClass for the webhook, setting to None enables dry-run
admissionControllerSideEffect: Unknown

# To restrict the kinds of operations and resources that are subject to OPA
# policy checks, see the settings below. By default, all resources and
# operations are subject to OPA policy checks.
admissionControllerRules:
  - operations: ["*"]
    apiGroups: ["*"]
    apiVersions: ["*"]
    resources: ["*"]

# Controls a PodDisruptionBudget for the OPA pod. Suggested use if having opa
# always running for admission control is important
podDisruptionBudget:
  enabled: false
  minAvailable: 1
# maxUnavailable: 1

# The helm Chart will automatically generate a CA and server certificate for
# the OPA. If you want to supply your own certificates, set the field below to
# false and add the PEM encoded CA certificate and server key pair below.
#
# WARNING: The common name name in the server certificate MUST match the
# hostname of the service that exposes the OPA to the apiserver. For example.
# if the service name is created in the "default" nanamespace with name "opa"
# the common name MUST be set to "opa.default.svc".
#
# If the common name is not set correctly, the apiserver will refuse to
# communicate with the OPA.
generateAdmissionControllerCerts: true
admissionControllerCA: ""
admissionControllerCert: ""
admissionControllerKey: ""

authz:
  # Disable if you don't want authorization.
  # Mostly useful for debugging.
  enabled: false

# Docker image and tag to deploy.
image: openpolicyagent/opa
imageTag: 0.15.1
imagePullPolicy: IfNotPresent

# Port to which the opa pod will bind itself
# NOTE IF you use a different port make sure it maches the ones in the readinessProbe
# and livenessProbe
port: 443

mgmt:
  enabled: true
  image: openpolicyagent/kube-mgmt
  imageTag: "0.10"
  imagePullPolicy: IfNotPresent
  extraArgs: []
  resources: {}
  data:
    enabled: true
  configmapPolicies:
# NOTE IF you use these, remember to update the RBAC rules below to allow
#      permissions to get, list, watch, patch and update configmaps
    enabled: true
    namespaces: [opa, kube-federation-scheduling-policy]
    requireLabel: true
  replicate:
# NOTE IF you use these, remember to update the RBAC rules below to allow
#      permissions to replicate these things
    cluster: []
#     - [group/]version/resource
    namespace: []
#     - [group/]version/resource
    path: kubernetes

# Log level for OPA ('debug', 'info', 'error') (app default=info)
logLevel: info

# Log format for OPA ('text', 'json') (app default=text)
logFormat: text

# Number of OPA replicas to deploy. OPA maintains an eventually consistent
# cache of policies and data. If you want high availability you can deploy two
# or more replicas.
replicas: 1

# To control how the OPA is scheduled on the cluster, set the affinity,
# tolerations and nodeSelector values below. For example, to deploy OPA onto
# the master nodes, 1 replica per node:
#
# affinity:
#   podAntiAffinity:
#     requiredDuringSchedulingIgnoredDuringExecution:
#     - labelSelector:
#         matchExpressions:
#         - key: "app"
#           operator: In
#           values:
#           - opa
#       topologyKey: "kubernetes.io/hostname"
# tolerations:
# - key: "node-role.kubernetes.io/master"
#   effect: NoSchedule
#   operator: Exists
# nodeSelector:
#   kubernetes.io/role: "master"
affinity: {}
tolerations: []
nodeSelector: {}

# To control the CPU and memory resource limits and requests for OPA, set the
# field below.
resources: {}

rbac:
  # If true, create & use RBAC resources
  #
  create: true
  rules:
    cluster:
    - apiGroups:
        - ""
      resources:
      - namespaces
      verbs:
      - get
      - list
      - watch
    - apiGroups:
        - ""
      resources:
      - configmaps
      verbs:
      - get
      - list
      - watch
      - patch
      - update

serviceAccount:
  # Specifies whether a ServiceAccount should be created
  create: true
  # The name of the ServiceAccount to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

# This proxy allows opa to make Kubernetes SubjectAccessReview checks against the
# Kubernetes API. You can get a rego function at github.com/open-policy-agent/library
sar:
  enabled: false
  image: lachlanevenson/k8s-kubectl
  imageTag: latest
  imagePullPolicy: IfNotPresent
  resources: {}

# To control the liveness and readiness probes change the fields below.
readinessProbe:
  httpGet:
    path: /health
    scheme: HTTPS
    port: 443
  initialDelaySeconds: 3
  periodSeconds: 5
livenessProbe:
  httpGet:
    path: /health
    scheme: HTTPS
    port: 443
  initialDelaySeconds: 3
  periodSeconds: 5

# Set a priorityClass using priorityClassName
# priorityClassName:

# Timeout for a webhook call in seconds.
# Starting in kubernetes 1.14 you can set the timeout and it is
# encouraged to use a small timeout for webhooks. If the webhook call times out, the request
# the request is handled according to the webhook'sfailure policy.
# timeoutSeconds: 20

securityContext:
  enabled: false
  runAsNonRoot: true
  runAsUser: 1

deploymentStrategy: {}
  # rollingUpdate:
  #   maxSurge: 1
  #   maxUnavailable: 0
  # type: RollingUpdate

---------------------

Step 2: At this point, I am able to test default policy (ingress-OK and ingress-bad) which works. All good so far.

Step 3: Now I want to implement my custom policy which is as follows:

apiVersion: v1
data:
  deprek8_check.rego: |-
    package kubernetes.admission

    import data.kubernetes.namespaces

    import input.request.object.metadata.annotations as annotations

    deny[msg] {
      input.apiVersion == "extensions/v1beta1"
      resources := ["Deployment"]
      input.kind == resources[_]
      msg := sprintf("%s/%s: API extensions/v1beta1 for %s is no longer served by default, use apps/v1 instead.", [input.kind, input.metadata.name, input.kind])
    }

    main = {
    	"apiVersion": "admission.k8s.io/v1beta1",
    	"kind": "AdmissionReview",
    	"response": response,
    }

    default response = {"allowed": true}

    response = {
    	"allowed": false,
    	"status": {"reason": reason},
    } {
    	reason := concat(", ", deny)
    	reason != ""
    }

kind: ConfigMap
metadata:
  name: deprek8-policies.rego
  namespace: opa
  labels:
    openpolicyagent.org/policy: rego

Step 4: I have verified that my custom policy gets loaded to opa:

~ curl -k -s https://localhost:8091/v1/policies | jq -r '.result[].raw'
package system

import data.kubernetes.admission

main = {
  "apiVersion": "admission.k8s.io/v1beta1",
  "kind": "AdmissionReview",
  "response": response,
}

default response = {"allowed": true}

response = {
    "allowed": false,
    "status": {
        "reason": reason,
    },
} {
    reason = concat(", ", admission.deny)
    reason != ""
}

package kubernetes.admission

import data.kubernetes.namespaces

import input.request.object.metadata.annotations as annotations

deny[msg] {
  input.apiVersion == "extensions/v1beta1"
  resources := ["Deployment"]
  input.kind == resources[_]
  msg := sprintf("%s/%s: API extensions/v1beta1 for %s is no longer served by default, use apps/v1 instead.", [input.kind, input.metadata.name, input.kind])
}

main = {
	"apiVersion": "admission.k8s.io/v1beta1",
	"kind": "AdmissionReview",
	"response": response,
}

default response = {"allowed": true}

response = {
	"allowed": false,
	"status": {"reason": reason},
} {
	reason := concat(", ", deny)
	reason != ""
}
package helm_kubernetes_quickstart

whitelist = [
	"*.dev.acmecorp.com",
	"*.test.acmecorp.com",
]

deny[msg] {
	input.request.kind.kind == "Ingress"
	input.request.operation == "CREATE"
	input.request.namespace == "opa-example"
	host := input.request.object.spec.rules[_].host
	not glob_match_one_of(whitelist, host)
	msg := sprintf("ingress host %q is invalid", [host])
}

glob_match_one_of(patterns, str) {
	glob.match(patterns[_], ["."], str)
}

main = {
	"apiVersion": "admission.k8s.io/v1beta1",
	"kind": "AdmissionReview",
	"response": response,
}

default response = {"allowed": true}

response = {
	"allowed": false,
	"status": {"reason": reason},
} {
	reason := concat(", ", deny)
	reason != ""
}

package test_helm_kubernetes_quickstart

import data.helm_kubernetes_quickstart

test_allow_outside_example_namespace {
    helm_kubernetes_quickstart.deny == set() with input as {
        "request": {
            "operation": "CREATE",
            "kind": {"kind": "Ingress"},
            "namespace": "default",
            "object": {
                "spec": {
                    "rules": [
                        {
                            "host": "payments.acmecorp.com",
                        },
                    ]
                }
            }
        }
    }
}

test_allow_inside_example_namespace {
    helm_kubernetes_quickstart.deny == set() with input as {
        "request": {
            "operation": "CREATE",
            "kind": {"kind": "Ingress"},
            "namespace": "opa-example",
            "object": {
                "spec": {
                    "rules": [
                        {
                            "host": "payments.dev.acmecorp.com",
                        },
                    ]
                }
            }
        }
    }
    helm_kubernetes_quickstart.deny == set() with input as {
        "request": {
            "operation": "CREATE",
            "kind": {"kind": "Ingress"},
            "namespace": "opa-example",
            "object": {
                "spec": {
                    "rules": [
                        {
                            "host": "payments.test.acmecorp.com",
                        },
                    ]
                }
            }
        }
    }
}

test_deny_inside_example_namespace {
    helm_kubernetes_quickstart.deny["ingress host \"payments.acmecorp.com\" is invalid"] with input as {
        "request": {
            "operation": "CREATE",
            "kind": {"kind": "Ingress"},
            "namespace": "opa-example",
            "object": {
                "spec": {
                    "rules": [
                        {
                            "host": "payments.acmecorp.com",
                        },
                    ]
                }
            }
        }
    }
}

Step 5: I have tested in the opa playground that the my custom policy works: https://play.openpolicyagent.org/p/mL0c2THp3L

Step 6: When I test my custom policy by deplying nginx, it seems that my custom policy is ignored

$ cat ~/workspace/nginx-bad.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
$ kubectl apply -f ~/workspace/nginx-bad.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
deployment.extensions/nginx-deployment configured

Can you please look into it and suggest if I am missing something.

Thanks, Suman