istio: Broken Multicluster Installation Task

Creating this issue for 1.7 bug handling for multicluster here. The new user guide is still in review so it’s a given that there’s some bugs in documenting this task. [User Task] https://github.com/istio/istio.io/pull/7787 [Preview] https://deploy-preview-7787--preliminary-istio.netlify.app/latest/docs/setup/install/multicluster/single-network/

I’ve run into following feature issues:

During initial configuration as well as while adding Remote/Primary cluster Generate the root CA and the intermediate CA for the cluster with the following command: make -f ${ISTIO}/tools/certs/Makefile ${CLUSTER_1}-cacerts-k8s

This command was earlier pointing to a Makefile in install/certs, but in 1.6 onwards the Makefile was changed and moved to tools/certs earlier the make command used to generate certs was ${NAME}-certs. I’m not sure what the new Make command is. Generating certs with Makefile from Istio 1.5 can be used to get by this issue.

Second and a more critical issue: Remote Cluster does not work correctly with Primary Cluster’s istiod. Steps to reproduce:

1. Deploy initial cluster
2. Try adding a primary cluster
3. Do the steps in verify your deployment.
4. Try deploying a sidecar injected helloworld app in remote cluster following the verify your deployment guide.

istio-proxy container for helloworld app in remote cluster never comes live.

Logs from istio-proxy in helloworld pod: Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected

Logs from istiod in remote:

2020-07-30T21:47:07.878563Z	info	ads	Push debounce stable[4] 5: 100.148341ms since last change, 172.94924ms since last push, full=true
2020-07-30T21:47:07.879276Z	info	ads	XDS: Pushing:2020-07-30T21:47:07Z/4 Services:9 ConnectedEndpoints:0
2020-07-30T21:47:08.479441Z	info	ads	Push debounce stable[5] 1: 100.190578ms since last change, 100.190344ms since last push, full=false
2020-07-30T21:47:08.479884Z	info	ads	XDS:EDSInc Pushing:2020-07-30T21:47:07Z/4 Services:map[istio-ingressgateway.istio-system.svc.cluster.local:{}] ConnectedEndpoints:0
2020-07-30T21:47:08.973286Z	info	ads	Push debounce stable[6] 1: 100.184855ms since last change, 100.184649ms since last push, full=false
2020-07-30T21:47:08.973680Z	info	ads	XDS:EDSInc Pushing:2020-07-30T21:47:07Z/4 Services:map[istio-egressgateway.istio-system.svc.cluster.local:{}] ConnectedEndpoints:0
2020-07-30T21:47:09.492411Z	info	ads	Full push, new service istio-egressgateway.istio-system.svc.cluster.local
2020-07-30T21:47:09.592716Z	info	ads	Push debounce stable[7] 1: 100.167655ms since last change, 100.167435ms since last push, full=true
2020-07-30T21:47:09.593216Z	info	ads	XDS: Pushing:2020-07-30T21:47:09Z/5 Services:9 ConnectedEndpoints:0
2020-07-30T21:47:10.772946Z	info	ads	Full push, new service istio-ingressgateway.istio-system.svc.cluster.local
2020-07-30T21:47:10.873281Z	info	ads	Push debounce stable[8] 1: 100.187272ms since last change, 100.187088ms since last push, full=true
2020-07-30T21:47:10.873898Z	info	ads	XDS: Pushing:2020-07-30T21:47:10Z/6 Services:9 ConnectedEndpoints:0
2020-07-30T21:47:12.048209Z	info	ads	Push Status: {}
2020-07-30T21:47:45.907413Z	info	ads	Push debounce stable[9] 1: 100.220431ms since last change, 100.220197ms since last push, full=true
2020-07-30T21:47:45.907990Z	info	ads	XDS: Pushing:2020-07-30T21:47:45Z/7 Services:9 ConnectedEndpoints:0
2020-07-30T21:47:52.048079Z	info	ads	Push Status: {}
2020-07-30T21:48:12.333042Z	info	Handle EDS endpoint: skip updating, service helloworld/sample has not been populated
2020-07-30T21:48:12.417848Z	info	ads	Push debounce stable[10] 1: 100.268112ms since last change, 100.267737ms since last push, full=true
2020-07-30T21:48:12.418618Z	info	ads	XDS: Pushing:2020-07-30T21:48:12Z/8 Services:8 ConnectedEndpoints:0
2020-07-30T21:48:22.048047Z	info	ads	Push Status: {}
2020-07-30T21:48:33.355048Z	info	ads	Push debounce stable[11] 2: 100.201299ms since last change, 111.044048ms since last push, full=true
2020-07-30T21:48:33.355550Z	info	ads	XDS: Pushing:2020-07-30T21:48:33Z/9 Services:9 ConnectedEndpoints:0
2020-07-30T21:48:35.104268Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=sample Name= (helloworld-v2-776f74c475-***** (actual name not yet known)) UID=88f6f23d-a956-4336-8442-ad9739938694 Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller e2a30f71-960c-4de1-b9aa-2d91c8a9619e [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2020-07-30T21:48:36.854586Z	info	ads	Push debounce stable[12] 1: 100.207151ms since last change, 100.206952ms since last push, full=false
2020-07-30T21:48:36.854688Z	info	ads	XDS:EDSInc Pushing:2020-07-30T21:48:33Z/9 Services:map[helloworld.sample.svc.cluster.local:{}] ConnectedEndpoints:0
2020-07-30T21:48:42.048110Z	info	ads	Push Status: {}

Logs from istiod in primary cluster:

2020-07-30T21:48:39.844186Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
2020-07-30T21:48:56.891548Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
2020-07-30T21:48:57.106406Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
2020-07-30T21:49:20.289474Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:49:25.948554Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:49:29.272108Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:49:36.932513Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
2020-07-30T21:49:39.340650Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
2020-07-30T21:49:43.261733Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:49:46.406920Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:49:48.011703Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:49:56.307653Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Node: Unauthorized
2020-07-30T21:50:17.567340Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:50:19.014094Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
2020-07-30T21:50:19.089760Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:50:19.634474Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:50:33.335838Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:50:33.830132Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
2020-07-30T21:50:45.651085Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Node: Unauthorized
2020-07-30T21:50:53.677884Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:51:05.080662Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
2020-07-30T21:51:07.219864Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to watch *v1.Pod: the server has asked for the client to provide credentials (get pods)
2020-07-30T21:51:08.375341Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Pod: Unauthorized
2020-07-30T21:51:10.986099Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Pod: Unauthorized
2020-07-30T21:51:11.003818Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
2020-07-30T21:51:15.953514Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Pod: Unauthorized
2020-07-30T21:51:19.596177Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:51:25.925143Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:51:27.028485Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Pod: Unauthorized
2020-07-30T21:51:34.501938Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Node: Unauthorized
2020-07-30T21:51:43.527114Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Unauthorized
2020-07-30T21:51:50.467311Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Pod: Unauthorized
2020-07-30T21:51:51.326362Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:51:59.750573Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Unauthorized
2020-07-30T21:52:00.372104Z	warn	serverca	Authentication failed for 10.60.8.2:51854: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:istio-system:istio-reader-service-account" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope. Authenticator ClientCertAuthenticator at index 2 got error: no verified chain is found. 
2020-07-30T21:52:04.525383Z	error	k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Node: Unauthorized