istio: Istio Authorization Policy not triggering checks - rbac_access_denied_matched_policy[default-deny-all-due-to-bad-CUSTOM-action]

Bug Description

Hi,

I have been trying to setup the Authorization policy sample for httpbin service using a HTTP ext-authz provider as described here: https://istio.io/latest/docs/tasks/security/authorization/authz-custom/

Here are my deployment and service declarations

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80

My istio mesh config for external aut is as below

mesh: |-
    extensionProviders:
    - name: "sample-ext-authz-http"
      envoyExtAuthzHttp:
        service: "ext-auth-node.platform.svc.cluster.local"
        port: "80"
        includeRequestHeadersInCheck: ["x-ext-authz"]

I can confirm that the service ext-auth-node.platform is accessible and return 200 on path / if it sees some valid JWT checks and 403 is the checks fail. I was able to curl this endpoint (for testing reacheability) from the proxy sidecar on the httpbin service

my Authorization Policy resource looks like so:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ext-authz
spec:
  selector:
    matchLabels:
      app: httpbin
  action: CUSTOM
  provider:
    # The provider name must match the extension provider defined in the mesh config.
    # You can also replace this with sample-ext-authz-http to test the other external authorizer definition.
    name: "sample-ext-authz-http"
  rules:
  # The rules specify when to trigger the external authorizer.
  - to:
    - operation:
        paths: ["/headers"]

HTTPbin service is running in the httpbin namespace, the ext-authz-node is running in platform namespace.

Expected:

When hitting the /headers service endpoint in httpbin, it should redirect the call to the ext-auth-node servcie, check the headers and then provide a 200 or 403 back to the envoy filter which in trun will decide on whethere or not to ALLOW or DENY

Actual: the ext-auth-node is not triggered as the envoy rbac logs are showing that the CUSTOM action specified is bad

when hitting the /httpbin/headers service endpoint the rbac:debug logs from the httpbin pod shows up the following

'x-envoy-peer-metadata-id', 'router~10.240.0.212~istio-ingressgateway-75bc568988-q8nft.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-envoy-original-path', '/httpbin/headers'
'x-b3-traceid', '563ac6a66b984cf4a04a166c06540002'
'x-b3-spanid', 'a04a166c06540002'
'x-b3-sampled', '1'
'x-envoy-internal', 'true'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/httpbin/sa/httpbin;Hash=ecfb7b313ffc26f819defd68f6a4dec6db1be720114b1e26b63877911d677957;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account'
, dynamicMetadata:
2022-09-13T04:14:16.765951Z     debug   envoy rbac      enforced denied, matched policy default-deny-all-due-to-bad-CUSTOM-action
[2022-09-13T04:14:16.765Z] "GET /httpbin/headers HTTP/1.1" 403 - rbac_access_denied_matched_policy[default-deny-all-due-to-bad-CUSTOM-action] - "-" 0 19 0 - "10.240.0.4" "PostmanRuntime/7.29.2" "94bbfaae-9edd-940d-9364-88da4ef9acac" "20.12.196.43" "-" inbound|80|| - 10.240.0.248:80 10.240.0.4:0 outbound_.8000_._.httpbin.httpbin.svc.cluster.local -

It seems to suggest that my custom action definition is bad. I have tried setting the paths to /httpbin/headers as well, but the RBAC policy refuses to identify the policy.

I though that maybe I am reading the service spec incorrectly and went through the AUthorization Policy spec here: https://istio.io/latest/docs/reference/config/security/authorization-policy/ and i guess mostly everything is in order.

Version

$ istioctl version
client version: 1.14.3
control plane version: 1.14.3
data plane version: 1.14.3 (10 proxies)

$ kubectl version --short
Client Version: v1.24.2
Kustomize Version: v4.5.4
Server Version: v1.21.14

$ helm version --short
v3.9.4+gdbc6d8e

Additional Information

No response