istio: Docker-desktop + SDS-auth values fails to start

Bug description

Just setting up Istio on Docker for Windows locally, fails, with

// etc
istio-system   0s          Warning   FailedMount                    pod/istio-ingressgateway-5c4f87768-csdbs       MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the server could not find the requested resource
istio-system   0s          Warning   FailedMount                    pod/istio-pilot-7576fc8fff-9sttl               MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the server could not find the requested resource
// etc

Expected behavior

Since https://istio.io/docs/setup/platform-setup/docker/ doesn’t mention any peculiarities and I was under the assumption that Citadel would generate these tokens automatically, it is a bug that istio-token is not generated. Here is one of the many configs that reference the token:

      - name: istio-token
        projected:
          sources:
          - serviceAccountToken:
              path: istio-token
              expirationSeconds: 43200
              audience: istio-c

in this case from Deployment istio-ingressgateway’s spec for istio-proxy.

Steps to reproduce the bug

Download istio-1.3.3 to Infrastructure/istio, then in a makefile:

# From https://istio.io/docs/setup/install/helm/#option-1-install-with-helm-via-helm-template

.PHONY: template
template:
	helm template ../../Infrastructure/istio/helm/istio-init \
		--name istio-init --namespace istio-system \
		> ./istio-init/k8s/base/template.yaml

	# if CNI is re-activated, remember to deploy it also
	helm template ../../Infrastructure/istio/helm/istio \
		--name istio --namespace istio-system \
		--set istio_cni.enabled=false \
		--set sidecarInjectorWebhook.enabled=true \
		--values ../../Infrastructure/istio/helm/istio/values-istio-sds-auth.yaml \
		--set pilot.traceSampling=100.0 \
		--set kiali.enabled=true \
		--set kiali.createDemoSecret=true \
		--set global.tracer.zipkin.address=jaeger-collector.monitoring:9411 \
		--set kiali.dashboard.jaegerURL=jaeger.afa.haf.se \
		> ./istio/k8s/base/template.yaml

# -- TEST aka Docker Desktop

.PHONY: deploy_init_test
deploy_init_test:
	kustomize build ./istio-init/k8s/test | kubectl apply -f -

.PHONY: deploy_istio_test
deploy_istio_test:
	kustomize build ./istio/k8s/test | kubectl apply -f -

deploy_test: template deploy_init_test deploy_istio_test

How was Istio installed?

Via the above;

  1. Pull down v1.3.3
  2. Template to new k8s cluster on docker for desktop with the above values
  3. Run make deploy_test until all CRDs allow instantiations (i.e. until there are no errors reported during installation)
  4. Look at kubectl get pods --all-namespaces or kubectl get all -n istio-system or kubectl get events -w --all-namespaces to see the “failed to fetch token: the server could not find the requested resource”

At this point, restarting Citadel or any other pod in the cluster doesn’t magically appear the istio-token secret, so the installation is stuck.

Environment where bug was observed (cloud vendor, OS, etc)

  • OS: Windows with Docker for Desktop/Windows

TL;DR If you just follow the instruction at https://istio.io/docs/setup/install/helm/#option-1-install-with-helm-via-helm-template but you enable the SDS/auth (strict mTLS) values file, you get this problem.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 45 (18 by maintainers)

Commits related to this issue

Most upvoted comments

In the next release we are going to support SDS without those flags passes

The flags are documented but not how to find out their values. @pitlv2109

This only applies when using manifest apply. It is not possible when doing generate, because that command does not connect to the kubernetes cluster and therefor cannot do automatic detection.

I understand, but I’m using this thing; code as config, where we place all our config in our repositories, to make deployments repeatable. Having the templates be smart is not in our best interest, and we’re not going to start versioning the istioctl in our git repo.

We want this to be smart:

  • the software we run inside k8s, and if that’s not possible
  • the operator that runs the software, and if that’s not possible,
  • the config-as-code via kustomize, and if that’s not possible,
  • I don’t use the software

How do I fix this with a “recommended” solution?

By “doesn’t work” do you mean it returns nothing? That means it is working – and indicates your cluster does not support the feature. The doc here could be a little more clear

Right, so that means I must use “first-party” JWTs then? Can’t Istio know this?

EDIT: this is what we ended up with

istioctl manifest generate --set values.global.jwtPolicy=first-party-jwt --filename istio-demo.yaml > k8s-1.5/istio-test/template.yaml

Yes, we have both a development environment and a GKE environment; but Docker for Desktop does not expose kubeadm configuration (that I can find), so applying the kubeadmConfigPatches seems like a hard sell to me. (I can’t apply it with kustomize and there’s no configuration section of Docker for Desktop available) As far as I can tell, prow doesn’t run on desktop, but on some linux lab cluster set up with kubeadm.

IMO if you require your users to change the kubernetes API server from its defaults, you’ve moved beyond those same users and have failed at the open source bit of the project. I really appreciate the work you’re putting in, but it has to be usable, even if “less secure on the development laptop”, which would be the effect of not requiring “trustworthy JWTs”. Don’t you agree?

I think the env is irrelevant here. If you’re running with out of the box OSS Kubernetes, feel free to use the flags we use here https://github.com/istio/istio/tree/master/prow/config for Istio testing.