istio: Missing upgrade note for 1.23.0 Ambient mode

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

Upgrading Istio Ambient mode from 1.22.2 to 1.23.0 and it seems there is a missing upgrade note in https://istio.io/latest/news/releases/1.23.x/announcing-1.23/upgrade-notes/

My Istio configuration details (I would paste my IstioOperator manifest but it’s on another network):

  • Profile: ambient
  • Istiod is deployed to istio-system
  • CNI and ztunnel pods are deployed to kube-system. We do not allow privileged pods in any other namespace.

My issue was nearly identical to https://github.com/istio/istio/issues/52057 except I encountered the error only after upgrading Istio and changing nothing else in my configuration. Thanks to @yardenshoham commenting on that issue, I figured out the fix. I had to configure the env var CA_TRUSTED_NODE_ACCOUNTS: "kube-system/ztunnel" for the pilot component in my config then everything worked smoothly.

The default value for this env var on the istiod container was istio-system/ztunnel on my Istio 1.23.0 deployment. I checked my most recent v1.22.2 deployment and that env var was set to istio-system/ztunnel,kube-system/ztunnel automatically. So either the documentation should have mentioned that this value would need to be configured appropriately when upgrading to 1.23.0, if this change was intended, or there is a regression where that env var isn’t being configured correctly anymore when deploying ztunnel to a different namespace.

Version

Istio v1.23.0
Kubernetes 1.30.2

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (11 by maintainers)

Most upvoted comments

oh @Phenix66 my bad - it should be

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    pilot:
      trustedZtunnelNamespace: <MY_ZT_NAMESPACE>

Because the IstioOperator indirects the Helm values, but (apparently) in several different ways. (component.pilot = no, values.pilot = yes)

(why do we have 6 ways to do one thing? IDK - I do apologize tho)

That worked perfectly, thank you!

Oh it is in the change notes: https://istio.io/latest/news/releases/1.23.x/announcing-1.23/change-notes/#installation

I don’t use the Helm chart however and nothing in that line tells me that previous default behavior changed. Based on the upgrade notes, I expected to be able to install Istio 1.23.0 using my current IstioOperator manifest and the istioctl command.

Agreed, fair. It could have been more explicit that because the default behavior was a problem, the default behavior has changed. Since ambient is still in beta, maintaining strict back compat is not a requirement, but the relnote (that I wrote) could have been worded more explicitly.

Are all of the various places it needs to be configured documented somewhere? I haven’t come across a complete list of the places updates are needed, I kind of just figured it out when I initially deployed Ambient mode on 1.22.x.

istioctl install --set values.pilot.trustedZtunnelNamespace=<MY_ZTUNNEL_NS> or helm install istiod --set pilot.trustedZtunnelNamespace=<MY_ZTUNNEL_NS> at install time should be all that is required.

Or in operator-resource-speak:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    pilot:
      trustedZtunnelNamespace: <MY_ZTUNNEL_NS>

Helm has ways to describe the available values that can be overriden for a particular chart (helm get values),

istioctl is mostly just using the Helm templates under the hood, and the Operator resource itself is a bit vestigial since the operator has been deprecated - I don’t think there’s a way to do the same there.

This is something generally we are trying to improve.