istio: JWT Token always fails with 403 in Istio 1.21.0

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

When I upgrade Istio using Istioctl from version 1.20.3 to 1.21.0 all requests to our api fail with http status code 403.

The following configuration files work correctly in version 1.20.3:

AuthorizationPolicy

apiVersion: "security.istio.io/v1"
kind: "RequestAuthentication"
metadata:
  name: "foo-jwt"
  namespace: foo
spec:
  selector:
    matchLabels:
      app: foo
  jwtRules:
    - issuer: "https://example.keycloak.com/realms/realm-name"
      jwksUri: "https://example.keycloak.com/realms/realm-name/protocol/openid-connect/certs"
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: require-jwt-foo
  namespace: foo
spec:
  selector:
    matchLabels:
      app: foo
  action: ALLOW
  rules:
    - from:
        - source:
            requestPrincipals: [ "https://example.keycloak.com/realms/realm-name/*" ]
      to:
        - operation:
            methods: [ "GET", "POST", "PUT", "OPTIONS", "DELETE", "PATCH" ]
            paths: [ "/api/v1/some-path/*" ]
      when:
        - key: request.auth.claims[emailVerified]
          values: [ "true" ]
        - key: request.auth.claims[roles]
          values: [ "admin", "user" ]

The previous configuration works flawlessly with Istio version 1.20.3.

As a test we apply the following AuthorizationPolicy without deleting the previous configuration and all the requests work again but it is because in the following configuration it overrides any validation of the token

apiVersion: "security.istio.io/v1"
kind: "AuthorizationPolicy"
metadata:
  name: "http-foo"
  namespace: foo
spec:
  selector:
    matchLabels:
      app: foo
  rules:
    - to:
        - operation:
            methods: ["GET", "POST", "PUT", "OPTIONS", "DELETE"]

Version

istioctl version
client version: 1.21.0
control plane version: 1.21.0
data plane version: 1.21.0 (71 proxies)

Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0-eks-c417bb3

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 22 (11 by maintainers)

Most upvoted comments

Please supply a config dump from Envoy to further triage (specifically jwt_authn filter and RBAC filter).

OK, I’ll send a fix to make the “aud” claim match as a list or a value. Thank you for the bug report!

@zirain This should be a release blocker, I’ll cherry pick to the release candidate.

Same issue here with AuthorizationPolicy spec defined as:

spec:
  action: DENY
  rules:
  - when:
    - key: request.auth.audiences
      notValues:
      - aud1
      - aud2
envoy config dump for v1.20.X
{
    "name": "envoy.filters.http.rbac",
    "typed_config": {
        "@type": "type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC",
        "rules": {
            "action": "DENY",
            "policies": {
                "ns[<redacted_ns>]-policy[<redacted_name>]-rule[0]": {
                    "permissions": [
                        {
                            "and_rules": {
                                "rules": [
                                    {
                                        "any": true
                                    }
                                ]
                            }
                        }
                    ],
                    "principals": [
                        {
                            "and_ids": {
                                "ids": [
                                    {
                                        "not_id": {
                                            "or_ids": {
                                                "ids": [
                                                    {
                                                        "metadata": {
                                                            "filter": "istio_authn",
                                                            "path": [
                                                                {
                                                                    "key": "request.auth.audiences"
                                                                }
                                                            ],
                                                            "value": {
                                                                "string_match": {
                                                                    "exact": "aud1"
                                                                }
                                                            }
                                                        }
                                                    },
                                                    {
                                                        "metadata": {
                                                            "filter": "istio_authn",
                                                            "path": [
                                                                {
                                                                    "key": "request.auth.audiences"
                                                                }
                                                            ],
                                                            "value": {
                                                                "string_match": {
                                                                    "exact": "aud2"
                                                                }
                                                            }
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    }
                                ]
                            }
                        }
                    ]
                }
            }
        },
        "shadow_rules_stat_prefix": "istio_dry_run_allow_"
    }
}
envoy config dump for v1.21.1
{
    "name": "envoy.filters.http.rbac",
    "typed_config": {
        "@type": "type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC",
        "rules": {
            "action": "DENY",
            "policies": {
                "ns[<redacted_ns>]-policy[<redacted_name>]-rule[0]": {
                    "permissions": [
                        {
                            "and_rules": {
                                "rules": [
                                    {
                                        "any": true
                                    }
                                ]
                            }
                        }
                    ],
                    "principals": [
                        {
                            "and_ids": {
                                "ids": [
                                    {
                                        "not_id": {
                                            "metadata": {
                                                "filter": "envoy.filters.http.jwt_authn",
                                                "path": [
                                                    {
                                                        "key": "payload"
                                                    },
                                                    {
                                                        "key": "aud"
                                                    }
                                                ],
                                                "value": {
                                                    "or_match": {
                                                        "value_matchers": [
                                                            {
                                                                "string_match": {
                                                                    "exact": "aud1"
                                                                }
                                                            },
                                                            {
                                                                "string_match": {
                                                                    "exact": "aud2"
                                                                }
                                                            }
                                                        ]
                                                    }
                                                }
                                            }
                                        }
                                    }
                                ]
                            }
                        }
                    ]
                }
            }
        },
        "shadow_rules_stat_prefix": "istio_dry_run_allow_"
    }
}

@kyessenov do you see any issue why this throws “RBAC: access denied” in v1.21.1

The issue for repeated values and not values is the difference in how JWT is matched between Envoy and Istio. I’ll send a patch for that. Re: iss difference, I’ll look further. Sorry about this, we don’t have a good coverage for this that would have caught it earlier.