istio: Istiod fails to issue certificate to a remote cluster
In the scenario like https://github.com/istio/istio/issues/22918. I encounter an error on k8s 1.18 like below:
Authentication failed: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: the service account authentication returns an error: [invalid bearer token, square/go-jose: error in cryptographic primitive]. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found.
Looking through the codes, I do not find problem as the cluster ID and remote client all are there. I also printed them out:
2020-05-15T08:10:46.403412Z info iris token eyJhbGciOiJSUzI1NiIsImtpZCI6IlBQUHVLQ0VMcDVyNHFtNDZfVVdtcFY1V1U0MHJnamtpNWhTVUpCTzM2OW8ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJpc3Rpby1zeXN0ZW0iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiaXN0aW8taW5ncmVzc2dhdGV3YXktc2VydmljZS1hY2NvdW50LXRva2VuLWs5eGdiIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImlzdGlvLWluZ3Jlc3NnYXRld2F5LXNlcnZpY2UtYWNjb3VudCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjcwNjA2YzE4LTNmYWMtNDAxZS1hMjllLWVlNjU0MDE5YTk2MiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDppc3Rpby1zeXN0ZW06aXN0aW8taW5ncmVzc2dhdGV3YXktc2VydmljZS1hY2NvdW50In0.SvyMALm-fAkCrifFm7mKhJ1jZWukt3hIEO_GI0FBNkDtpP3Xs3XEAFdZepVINZTJW_7J8O4AFb2ADwcfG9Ce2WpysdgZigl71QsSOzy8uSCjNIaiczPiQE5QICAbBRMWSSRgkf4kj9YnPoZO8dEDh-Oo0CQLBfLc697T8SELxACkODQd6QZo4Jnhbp8mOry953IZ5Y-Lk2fkLjAefOgkDKHEEsL33DcMPYMQSRwiY5bo19aCMe5baRf6PUMBvM6X9wawB8SVA_WKqaglcq4KewlAdLdUjT7ZkP10MQgkOkBZi73dq9l23ykvode61eFU1gxJ9RtUpPHU6wvB1BKbkw
2020-05-15T08:10:46.403476Z info iris cluster id: remotecluster
2020-05-15T08:10:46.403486Z info iris incoming cluster id: remotecluster
2020-05-15T08:10:46.403493Z info iris current cluster id: maincluster
2020-05-15T08:10:46.403497Z info iris remote client is not nil
2020-05-15T08:10:46.406892Z warn serverca Authentication failed: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: the service account authentication returns an error: [invalid bearer token, square/go-jose: error in cryptographic primitive]. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found.
I compared the token from log with the one in the ingress gatewway pod and they are exactly the same. looks like :https://github.com/kubernetes/kubernetes/issues/72026 is related. But no other clue.
@myidpt @hzxuzhonghu @linsun Appreciated if you can share some lights here.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 19 (19 by maintainers)
One way to narrow down the issue scope is to verify your token by hand. https://jpweber.io/blog/a-look-at-tokenrequest-api/
Confirmed that if I modify the main cluster’s istio-sidecar-injector cm, and restart istiod afterwards, then deploy sleep pod on the remote cluster with auto sidecar injector, the pod will come up nicely. @hzxuzhonghu this confirms your change is okay but require us to configure the sidecar with proper cluster id in remote cluster.