istio: Istio 1.6.3: ingressgateway JWT refused - error “Proto constraint validation failed” using Keycloak

Bug description I’m trying to setup this RequestAuthentication.

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: "h-ingress-jwt"
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "http://192.168.202.21:8000/auth/realms/istio"
    jwksUri: "http://192.168.202.21:8000/auth/realms/istio/protocol/openid-connect/certs"

In the ingressgateway logs I see this error:

2020-07-09T09:36:50.669693Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 7 successful, 0 rejected; lds updates: 0 successful, 7 rejected
2020-07-09T09:36:52.669736Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 7 successful, 0 rejected; lds updates: 0 successful, 7 rejected
2020-07-09T09:36:54.670022Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 7 successful, 0 rejected; lds updates: 0 successful, 7 rejected
2020-07-09T09:36:55.068148Z     warning envoy config    [external/envoy/source/common/config/grpc_subscription_impl.cc:101] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8443: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "http://192.168.202.21:8000/auth/realms/istio"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "http://192.168.202.21:8000/auth/realms/istio"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}

0.0.0.0_8080: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "http://192.168.202.21:8000/auth/realms/istio"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "http://192.168.202.21:8000/auth/realms/istio"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}


2020-07-09T09:36:55.706732Z     warning envoy config    [external/envoy/source/common/config/grpc_subscription_impl.cc:101] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8443: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "http://192.168.202.21:8000/auth/realms/istio"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "http://192.168.202.21:8000/auth/realms/istio"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}

0.0.0.0_8080: Proto constraint validation failed (JwtAuthenticationValidationError.Providers[key]: ["embedded message failed validation"] | caused by JwtProviderValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): providers {
  key: "origins-0"
  value {
    issuer: "http://192.168.202.21:8000/auth/realms/istio"
    local_jwks {
      inline_string: ""
    }
    payload_in_metadata: "http://192.168.202.21:8000/auth/realms/istio"
  }
}
rules {
  match {
    prefix: "/"
  }
  requires {
    requires_any {
      requirements {
        provider_name: "origins-0"
      }
      requirements {
        allow_missing {
        }
      }
    }
  }
}


2020-07-09T09:36:56.669676Z     warn    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 9 successful, 0 rejected; lds updates: 0 successful, 9 rejected

If I understand correctly, the ingressgatewaway didn’t receive a reply from http://192.168.202.21:8000/auth/realms/istio but ececuting a curl from the ingressgateway pod I receive some data in response:

kubectl exec -it istio-ingressgateway-66cc54b468-pkmd7 -n istio-system -- curl  http://192.168.202.21:8000/auth/realms/istio
{"realm":"istio","public_key":"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwC48QVPBIDZ9ormmLqlwjZDCrur1HOyQOBsIRlWr6ieMQTNqqulWtRlmv+IcGScsXvZtotNfwJlRxGvYwoK/gg1J7gT4EzDwxVs7FKN7YktDKcT8HyYuAIXFZta15jxC+pOhJseZVQDpWPLjv5dn1vGEMe400v4uFTjqC1mZJohoPcNA5Q3IjwPC72LvJ0hGGPLzpv6gmMg/XNw3xK/z+5dctIH0PaOM+hh7wcDw45j32gEOyEHHxDWJzlJl1s4ckwVLCg+Wmof7UyW/aaqqoOqGxdjnNHz/nuYO61Rp7z1sHBn9/GMsJxZwlsxjCV23Vabccu1Iswjkn97sXZDtfQIDAQAB","token-service":"http://192.168.202.21:8000/auth/realms/istio/protocol/openid-connect","account-service":"http://192.168.202.21:8000/auth/realms/istio/account","tokens-not-before":0}

I receive the same response isuing curl from the istiod pod

Triyng to access any service gives a 401 error

sysop@hdev:~/software$ curl --insecure -L -X POST 'https://192.168.202.21:8443/auth/realms/istio/protocol/openid-connect/token' \
> -H 'Content-Type: application/x-www-form-urlencoded' \
> --data-urlencode 'client_id=cistio' \
> --data-urlencode 'grant_type=password' \
> --data-urlencode 'client_secret=d74ae23c-4b7b-4f88-8876-4a0456e6f832' \
> --data-urlencode 'scope=openid' \
> --data-urlencode 'username=test' \
> --data-urlencode 'password=test' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3332  100  3204  100   128  19418    775 --:--:-- --:--:-- --:--:-- 20317
{
  "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJtS3hqTHFWVnFTSThZRzJWUFZBZEhmQkljN24zZ0NXS1UyMTZGSTU1NGN3In0.eyJleHAiOjE1OTQ5MTgwMDcsImlhdCI6MTU5NDkxNzcwNywianRpIjoiNGM4ZTNmMmYtNTQxOC00NDVmLWE3ODQtNjI5OTZiMjczMWIyIiwiaXNzIjoiaHR0cHM6Ly8xOTIuMTY4LjIwMi4yMTo4NDQzL2F1dGgvcmVhbG1zL2lzdGlvIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6IjdmNzdiOWI0LThjNzgtNDM4MS04NTIzLTAwZDc5MTgzNzZhZiIsInR5cCI6IkJlYXJlciIsImF6cCI6ImNpc3RpbyIsInNlc3Npb25fc3RhdGUiOiJiZGY5NTgxZi0wYjA3LTQxM2YtOTMyOC00OTcwMWM5ODQ5NjgiLCJhY3IiOiIxIiwiYWxsb3dlZC1vcmlnaW5zIjpbIiR7YXV0aEJhc2VVcmx9Il0sInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJlZGl0b3IiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJvcGVuaWQgcHJvZmlsZSBlbWFpbCIsImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwibmFtZSI6InRlc3QgdGVzdCIsInByZWZlcnJlZF91c2VybmFtZSI6InRlc3QiLCJnaXZlbl9uYW1lIjoidGVzdCIsImZhbWlseV9uYW1lIjoidGVzdCIsImVtYWlsIjoidGVzdEBoLm5ldCJ9.LzQfUH16Q52H3GzM1NCwaB1UfF_O6o4VqH6gWc4wMatng2xXxo63ztuihpaI4zxI91dq_SY2u5drbZPh6crC0Ie3_tDrymKt3uOwvu_wTR1AiI_RhoWgSQ648ovm3XIvl0zeNzfcCC2168-1__5bRud8UYtVyttTqeDMgI3VA3JLWJ23m3VhA9mIdyx0KPmSeDQ3mOqR3o4vUYAPbDRifz6GYMRvHtqRZp5mma_uJbKjPA3XSki3966FMeU92Rx2FFAg6cbHwclJt_zr7nyIinD60ItjQwhZBybpnzY68GmpH2TlofC6LKobFDDAFPufRzkvjhI0rfq0JdvNg54EIA",
  "expires_in": 300,
  "refresh_expires_in": 1800,
  "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI5MTA3YTUwMy1kMjNhLTRhYjctODc0OS01NmMyMDY4OWY3NzAifQ.eyJleHAiOjE1OTQ5MTk1MDcsImlhdCI6MTU5NDkxNzcwNywianRpIjoiNjAzYmQ2Y2UtNzliOC00MGU2LTgwYWEtMTk2ODU5MDdiNjk1IiwiaXNzIjoiaHR0cHM6Ly8xOTIuMTY4LjIwMi4yMTo4NDQzL2F1dGgvcmVhbG1zL2lzdGlvIiwiYXVkIjoiaHR0cHM6Ly8xOTIuMTY4LjIwMi4yMTo4NDQzL2F1dGgvcmVhbG1zL2lzdGlvIiwic3ViIjoiN2Y3N2I5YjQtOGM3OC00MzgxLTg1MjMtMDBkNzkxODM3NmFmIiwidHlwIjoiUmVmcmVzaCIsImF6cCI6ImNpc3RpbyIsInNlc3Npb25fc3RhdGUiOiJiZGY5NTgxZi0wYjA3LTQxM2YtOTMyOC00OTcwMWM5ODQ5NjgiLCJzY29wZSI6Im9wZW5pZCBwcm9maWxlIGVtYWlsIn0.nnmj_f2QGnj5In3Ir_r8k1OkRbB6r4ztz4I5XL81O-g",
  "token_type": "bearer",
  "id_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJtS3hqTHFWVnFTSThZRzJWUFZBZEhmQkljN24zZ0NXS1UyMTZGSTU1NGN3In0.eyJleHAiOjE1OTQ5MTgwMDcsImlhdCI6MTU5NDkxNzcwNywiYXV0aF90aW1lIjowLCJqdGkiOiI4YTYwMzExNC00ZDU0LTQzZDMtOWU2Yy01YjZhYzRiYzI0NDYiLCJpc3MiOiJodHRwczovLzE5Mi4xNjguMjAyLjIxOjg0NDMvYXV0aC9yZWFsbXMvaXN0aW8iLCJhdWQiOiJjaXN0aW8iLCJzdWIiOiI3Zjc3YjliNC04Yzc4LTQzODEtODUyMy0wMGQ3OTE4Mzc2YWYiLCJ0eXAiOiJJRCIsImF6cCI6ImNpc3RpbyIsInNlc3Npb25fc3RhdGUiOiJiZGY5NTgxZi0wYjA3LTQxM2YtOTMyOC00OTcwMWM5ODQ5NjgiLCJhY3IiOiIxIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJuYW1lIjoidGVzdCB0ZXN0IiwicHJlZmVycmVkX3VzZXJuYW1lIjoidGVzdCIsImdpdmVuX25hbWUiOiJ0ZXN0IiwiZmFtaWx5X25hbWUiOiJ0ZXN0IiwiZW1haWwiOiJ0ZXN0QGgubmV0In0.L9t7Iv5VG4abgupupuPxhFAXDIUwbrMBikcWcW8fnZqV5DKSfjm0yItk-CP9kOxQuO7wlg6jlWd_eX-sj1mrDCJ88DXw0LFitF9dT6eRvQay4RHdmKAvu6GM3uePodMHcZQKT8ClEUZ0t4QzZm5Uy0mZ6WPuTT7kWxGPw-F2t7I1qA3CXYYcH_C8lZl5Bp5vOM57FrI8EvBNzDxq1rwavHX1Ilz8Sf4eSH0BiVc99zvFe6mJYNbaWGTsblIqaSw8jv1qrHQjwnhseaW5PwZnR3o4T-9IW9g4pikTL5kNk_Ya6IIECYsxdcxvQdt3B-nBxSG0rUyzxlzrFy29vco0Hg",
  "not-before-policy": 0,
  "session_state": "bdf9581f-0b07-413f-9328-49701c984968",
  "scope": "openid profile email"
}
sysop@hdev:~/software$ export TOKEN=eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJtS3hqTHFWVnFTSThZRzJWUFZBZEhmQkljN24zZ0NXS1UyMTZGSTU1NGN3In0.eyJleHAiOjE1OTQ5MTgwMDcsImlhdCI6MTU5NDkxNzcwNywianRpIjoiNGM4ZTNmMmYtNTQxOC00NDVmLWE3ODQtNjI5OTZiMjczMWIyIiwiaXNzIjoiaHR0cHM6Ly8xOTIuMTY4LjIwMi4yMTo4NDQzL2F1dGgvcmVhbG1zL2lzdGlvIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6IjdmNzdiOWI0LThjNzgtNDM4MS04NTIzLTAwZDc5MTgzNzZhZiIsInR5cCI6IkJlYXJlciIsImF6cCI6ImNpc3RpbyIsInNlc3Npb25fc3RhdGUiOiJiZGY5NTgxZi0wYjA3LTQxM2YtOTMyOC00OTcwMWM5ODQ5NjgiLCJhY3IiOiIxIiwiYWxsb3dlZC1vcmlnaW5zIjpbIiR7YXV0aEJhc2VVcmx9Il0sInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJlZGl0b3IiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJvcGVuaWQgcHJvZmlsZSBlbWFpbCIsImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwibmFtZSI6InRlc3QgdGVzdCIsInByZWZlcnJlZF91c2VybmFtZSI6InRlc3QiLCJnaXZlbl9uYW1lIjoidGVzdCIsImZhbWlseV9uYW1lIjoidGVzdCIsImVtYWlsIjoidGVzdEBoLm5ldCJ9.LzQfUH16Q52H3GzM1NCwaB1UfF_O6o4VqH6gWc4wMatng2xXxo63ztuihpaI4zxI91dq_SY2u5drbZPh6crC0Ie3_tDrymKt3uOwvu_wTR1AiI_RhoWgSQ648ovm3XIvl0zeNzfcCC2168-1__5bRud8UYtVyttTqeDMgI3VA3JLWJ23m3VhA9mIdyx0KPmSeDQ3mOqR3o4vUYAPbDRifz6GYMRvHtqRZp5mma_uJbKjPA3XSki3966FMeU92Rx2FFAg6cbHwclJt_zr7nyIinD60ItjQwhZBybpnzY68GmpH2TlofC6LKobFDDAFPufRzkvjhI0rfq0JdvNg54EIA
sysop@hdev:~/software$ 
sysop@hdev:~/software$ echo -e "\n\nWITHOUT token"


WITHOUT token
sysop@hdev:~/software$ curl --insecure https://www.h.net -s -o /dev/null -w "%{http_code}\n"
200
sysop@hdev:~/software$ echo -e "\n\nwith INVALID token"


with INVALID token
sysop@hdev:~/software$ curl --insecure --header "Authorization: Bearer pippo" https://www.h.net -s -o /dev/null -w "%{http_code}\n"
401
sysop@hdev:~/software$ echo -e "\n\nWITH token"


WITH token
sysop@hdev:~/software$ curl --insecure --header "Authorization: Bearer $TOKEN" https://www.h.net -s -o /dev/null -w "%{http_code}\n"
401
sysop@hdev:~/software$ 

What I’m doing wrong?

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure [ ] Docs [ ] Installation [ ] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ x] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Affected features (please put an X in all that apply)

[ ] Multi Cluster [ x] Virtual Machine [ ] Multi Control Plane

Expected behavior Obtain 200 return code accessing with jwt Using demo jwt with this RequestAuthentication

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: "h-ingress-jwt"
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: "testing@secure.istio.io"
    jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.6/security/tools/jwt/samples/jwks.json"

---

I obtain a 200 return code for valid jwt

sysop@hdev:~/software/hproject$ export TOKEN=$(curl https://raw.githubusercontent.com/istio/istio/release-1.6/security/tools/jwt/samples/demo.jwt -s)
sysop@hdev:~/software/hproject$ echo -e "\n\nWITHOUT token"


WITHOUT token
sysop@hdev:~/software/hproject$ curl --insecure https://www.h.net -s -o /dev/null -w "%{http_code}\n"
200
sysop@hdev:~/software/hproject$ echo -e "\n\nwith INVALID token"


with INVALID token
sysop@hdev:~/software/hproject$ curl --insecure --header "Authorization: Bearer pippo" https://www.h.net -s -o /dev/null -w "%{http_code}\n"
401
sysop@hdev:~/software/hproject$ echo -e "\n\nWITH token"


WITH token
sysop@hdev:~/software/hproject$ curl --insecure --header "Authorization: Bearer $TOKEN" https://www.h.net -s -o /dev/null -w "%{http_code}\n"
200
sysop@hdev:~/software/hproject$ 

Steps to reproduce the bug

• install a Keycloak server in the cluster via helm chart and expose using loadbalancer
• create a sealm “istio”, a client “cistio” and a user named “test”
• apply the first RequestAuthentication and verify jwt is not accepted

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

sysop@hdev:~/software/hproject$ istioctl version --remote
client version: 1.6.3
control plane version: 1.6.3
data plane version: 1.6.3 (8 proxies)
sysop@hdev:~/software/hproject$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4+k3s1", GitCommit:"97b7a0e9df2883f08028fb7171c1e62fc1899a0c", GitTreeState:"clean", BuildDate:"2020-06-18T01:30:45Z", GoVersion:"go1.13.11", Compiler:"gc", Platform:"linux/amd64"}
sysop@hdev:~/software/hproject$ helm version
version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}
sysop@hdev:~/software/hproject$ 

How was Istio installed?

istioctl manifest apply --set profile=demo

Environment where bug was observed (cloud vendor, OS, etc)

Framework:

• 4 KVM virtual machines with ubuntu server 20.04
• K3S kubernetes distribution (1 master 3 workers)
• Istio 1.6.3 (ugraded from 1.6.1, upgraded from 1.6.0)

Additionally, please consider attaching a cluster state archive by attaching the dump file to this issue. istio-dump.tar.gz 20200712-logs-from-discovery-in-istiod-777dc7dc48-rxf45.txt