istio: Istio sidecar injection failing with error - MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found

This issue needs to be re-opened. It definitely can still occur in all versions

I am facing the same issue with 1.5.1 and doing a restart of the istiod deployment is a valid workaround in the meantime:

kubectl -n istio-system rollout restart deployment istiod

Is everyone facing this issue on AKS? Seems 4 of the people that have reported it are on azure?

Still facing the same issue in 1.5.6

I have the same issue when I upgrade istio from 1.5.0 (installed by apply helm generated yaml files) to 1.5.4 by apply istioctl manifest generate yaml file.

I have both a fix and a reproducer. Fix: #23783

To reproduce, modify the istio-leader configmap and change holder identity to something else. After a minute or so everything will be broken

I got some thought from the aboved explain. So I check the istio-leader.

When I see the configmap istio-leader in the istio-system, I found the clue. the old pilot are still the istio-leader, so the new istiod will not became the namespace controller, so the configmap istio-ca-root-cert was not created in the namespace that was injected.

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"istio-pilot-9979d9b5c-bw7sw","leaseDurationSeconds":30,"acquireTime":"2020-05-29T01:36:04Z","renewTime":"2020-05-29T02:50:45Z","leaderTransitions":0}'
  creationTimestamp: "2020-05-29T01:36:04Z"
  name: istio-leader
  namespace: istio-system
  resourceVersion: "41583"
  selfLink: /api/v1/namespaces/istio-system/configmaps/istio-leader
  uid: 4842a8ce-fb32-4224-91e2-9e2e51acf86b

In order to fix this, we shoud make the new istiod become the istio-leader, so you can delete the old istio-pilot, or delete the istio-leader configmap in the istio-system, and then check if the new istiod is become the new leader

Istio does not yet support ARM which would explain the raspberry pi one

On Sun, Mar 20, 2022, 11:32 AM Greg @.***> wrote:

Same issue for me on a 4-node RasPi-4 cluster with Istio 1.13.2

— Reply to this email directly, view it on GitHub https://github.com/istio/istio/issues/22463#issuecomment-1073307268, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXMH24XEX3LGC3TUIR3VA5VKDANCNFSM4LTK3Z6A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

meet same issue in 1.9.0

I have both a fix and a reproducer. Fix: https://github.com/istio/istio/pull/23783

To reproduce, modify the istio-leader configmap and change holder identity to something else. After a minute or so everything will be broken

I am also getting this error using EKS 1.21 with Istio 1.11.5. Everything works fine, but this error is causing concern for the developers on our platform.

@dkirrane that is a difference issue, root cause is Istiod not starting up. Probably because of https://preliminary.istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens

There are a few PRs merged for this issue (e.g., https://github.com/istio/istio/pull/22598 and https://github.com/istio/istio/pull/22606). Does this issue occur for an Istio binary with these PRs?

Meanwhile, thanks @emedina for sharing a workaround, which restarts the istiod deployment through “kubectl -n istio-system rollout restart deployment istiod”.

@KIRY4 just update to 1.5.4 and it’s fixed

For docker-desktop, demo

For docker-desktop, trustworthy JWTs are still not enabled. Editing the API server pod config with:

    - --service-account-key-file=/run/config/pki/sa.pub
+    - --service-account-signing-key-file=/run/config/pki/sa.key
+    - --service-account-issuer=api
+    - --api-audiences=api

can’t be done as it’s overwritten by the Docker Desktop application continuously.

However, since SDS moved into istiod now, it can’t be shut off; despite configuring the IstioOperator CRD like so:

        sds:
          enabled: false
# and further down
      sds:
        enabled: false
        token:
          aud: istio-ca
        udsPath: ""

For GKE, demo

This error is new:

istiod-668dd9d86b-57nzg discovery {“level”:“info”,“time”:“2020-04-28T13:37:05.272378Z”,“scope”:“validationController”,“msg”:“Reconcile(enter): retry dry-run creation of invalid config”}

But then there’s a new port that needs to be opened from GKE master -> 15017, or you’ll get timeouts when deploying VirtualServices.

On the up-side, all tokens are generated and we got it running!!! 🔆

Hi,

We are facing the same issue with 1.5.2 on AKS, our pods are stuck on the Init phase:

Events:
  Type     Reason       Age                 From                                      Message
  ----     ------       ----                ----                                      -------
  Normal   Scheduled    42m                 default-scheduler                         Successfully assigned nto-payment/customer-app-5c5ff5c5f-hxs9f to aks-default-16109932-vmss000000
  Warning  FailedMount  11m (x23 over 42m)  kubelet, aks-default-16109932-vmss000000  MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found
  Warning  FailedMount  98s (x18 over 40m)  kubelet, aks-default-16109932-vmss000000  Unable to mount volumes for pod "customer-app-5c5ff5c5f-hxs9f_nto-payment(4b227c43-d6dd-42f4-aac5-78afef26c8e0)": timeout expired waiting for volumes to attach or mount for pod "nto-payment"/"customer-app-5c5ff5c5f-hxs9f". list of unmounted volumes=[istiod-ca-cert]. list of unattached volumes=[default-token-h5c9m istio-envoy podinfo istio-token istiod-ca-cert]

using this IstioOperator spec:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: default
  components:
    policy:
      enabled: true
    sidecarInjector:
      enabled: true
    citadel:
      enabled: true
    telemetry:
      enabled: true
  addonComponents:
    prometheus:
      enabled: false
  values:
    global:
      disablePolicyChecks: false
    telemetry:
      v1:
        enabled: true
      v2:
        enabled: false

started working again after restarting the istiod deployment and our app pods.