istio: Ubuntu 24.04 + Vanilla k8s + Calico + 1.23 Ambient - istio-cni failing to start on 2/3 nodes

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

Node are set up 100% identical save for a couple minor hardware differences on one, but two are 100% identical and one is working and one is failing.

Installed “vanilla k8s” with kubeadm, installed calico as per docs, cluster is running fine. Following the Ambient install guide and failing at the second step, installing istio-cni from helm (though I’m having the same issue with istioctl).

Two of the pods, on the same two nodes every time, are dying and staying in a backoff loop with -

Error: failed to create ambient nodeagent service: error configuring iptables: binary iptables-save not found in path: exit status 127

Yet they’re using the exact same image as the node it is starting successfully on, and the systems are configured the same and have iptables installed and iptables-save available (not sure if it’s referring to within a container or on the host tbh).

Troubleshooting for a full day now, and all I can come up with is ‘bug’

Version

istioctl version -
Istio is not present in the cluster: no running Istio pods in namespace "istio-system"
client version: 1.23.0

kubectl version -
Client Version: v1.30.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.3

helm version --short -
v3.15.4+gfa9efb0

Additional Information

Target cluster context: vcluster

Running with the following config:

istio-namespace: istio-system full-secrets: false timeout (mins): 30 include: { } exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage } end-time: 2024-08-21 21:37:36.725950598 -0400 EDT

Cluster endpoint: https://vcluster-vip.mydomain.tld:6444 CLI version: version.BuildInfo{Version:“1.23.0”, GitRevision:“d0ca037e58f90f58b38b33c5769fec25b6bfa9f3”, GolangVersion:“go1.22.6”, BuildStatus:“Clean”, GitTag:“1.23.0”}

The following Istio control plane revisions/versions were found in the cluster: The following proxy revisions/versions were found in the cluster:

Fetching logs for the following containers:

calico-apiserver/calico-apiserver/calico-apiserver-85b898779c-779t4/calico-apiserver calico-apiserver/calico-apiserver/calico-apiserver-85b898779c-tdwxh/calico-apiserver calico-system/calico-kube-controllers/calico-kube-controllers-77b9c68f4-96gjf/calico-kube-controllers calico-system/calico-node/calico-node-26pk8/calico-node calico-system/calico-node/calico-node-4rz7x/calico-node calico-system/calico-node/calico-node-g665r/calico-node calico-system/calico-typha/calico-typha-7854c6d879-2xdgf/calico-typha calico-system/calico-typha/calico-typha-7854c6d879-mq872/calico-typha calico-system/csi-node-driver/csi-node-driver-4fcwh/calico-csi calico-system/csi-node-driver/csi-node-driver-4fcwh/csi-node-driver-registrar calico-system/csi-node-driver/csi-node-driver-rslkr/calico-csi calico-system/csi-node-driver/csi-node-driver-rslkr/csi-node-driver-registrar calico-system/csi-node-driver/csi-node-driver-v8hsn/calico-csi calico-system/csi-node-driver/csi-node-driver-v8hsn/csi-node-driver-registrar istio-system/istio-cni-node/istio-cni-node-dw5wb/install-cni istio-system/istio-cni-node/istio-cni-node-jqqgj/install-cni istio-system/istio-cni-node/istio-cni-node-r8wvj/install-cni tigera-operator/tigera-operator/tigera-operator-77f994b5bb-mhv54/tigera-operator

Fetching Istio control plane information from cluster.

Fetching CNI logs from cluster.

Running Istio analyze on all namespaces and report as below: Analysis Report: Info [IST0102] (Namespace calico-apiserver) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace calico-apiserver istio-injection=enabled’ to enable it, or ‘kubectl label namespace calico-apiserver istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace calico-system) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace calico-system istio-injection=enabled’ to enable it, or ‘kubectl label namespace calico-system istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace default istio-injection=enabled’ to enable it, or ‘kubectl label namespace default istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kube-node-lease) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kube-node-lease istio-injection=enabled’ to enable it, or ‘kubectl label namespace kube-node-lease istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kube-public) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kube-public istio-injection=enabled’ to enable it, or ‘kubectl label namespace kube-public istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kube-system) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kube-system istio-injection=enabled’ to enable it, or ‘kubectl label namespace kube-system istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace tigera-operator) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace tigera-operator istio-injection=enabled’ to enable it, or ‘kubectl label namespace tigera-operator istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0118] (Service calico-apiserver/calico-api) Port name apiserver (port: 443, targetPort: 5443) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service calico-system/calico-kube-controllers-metrics) Port name metrics-port (port: 9094, targetPort: 9094) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service calico-system/calico-typha) Port name calico-typha (port: 5473, targetPort: calico-typha) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service kube-system/kube-dns) Port name metrics (port: 9153, targetPort: 9153) doesn’t follow the naming convention of Istio port. Creating an archive at /home/me/bug-report.tar.gz. Time used for creating the tar file is 143.414681ms. Cleaning up temporary files in /tmp/bug-report. Done.

bug-report.tar.gz

Please note the cluster is named vcluster, because I intend to install vcluster on it at some point, but it is not yet installed, and this is not a vcluster, it’s vanilla k8s running on Ubuntu VMs running on Proxmox.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 20 (9 by maintainers)

Most upvoted comments

I’ve got the same issue with Istio version 1.23.2, deployed in a Lima VM (KVM) with Kind cluster.

Deploying Istio with variant: debug works.

At the top, you can see that it is a KVM. On the left, installing with variant: distroless. On the right, the error log in the istio-cni-node pod

image

This does not work:

istioctl install -y --set profile=ambient --set meshConfig.accessLogFile=/dev/stdout --set values.global.variant=distroless

this works:

istioctl install -y --set profile=ambient --set meshConfig.accessLogFile=/dev/stdout --set values.global.variant=debug

Sorry, I mean the node image - what distro/kernel/glib/etc.

Ah!

During the last round of failures I was running Ubuntu Server 22.04, I can spin up a VM later to check what the kernel/glib versions are on that.

Earlier attempts and what I’ve since reverted to were Ubuntu Server 24.04 minimal -

ldd (Ubuntu GLIBC 2.39-0ubuntu8.3) 2.39
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
Linux vcluster-01 6.8.0-41-generic #41-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug  2 20:41:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Yet they’re using the exact same image as the node it is starting successfully on, and the systems are configured the same and have iptables installed and iptables-save available (not sure if it’s referring to within a container or on the host tbh).

Our istio base image ships both variants, and for ambient we are only running from containers, so we attempt to select the correct in-container binary based on whether the host node’s (nft|legacy) tables already have rules in them.

if we can’t “tiebreak” between nft/legacy by checking rulecount, we always fall back to the local default symlink e…g iptables/iptables-save).

My suspicion is that on some of your nodes, you have nft rules on the node already when we come up, which is why istio-cni-node-r8wvj for instance picks nft, and doesn’t have a problem.

The other nodes probably don’t have any rules in either legacy or nft tables, at the time we check, so we fall back to the local default symlink.

(in this scenario it would be better to just prefer nft, which would mask your problem here, as mentioned over here: https://github.com/istio/istio/issues/52357#issuecomment-2253300746)

You could confirm this theory (if you wish, no worries if not, already a great bug report) by installing istio-cni with debug logging enabled (--set global.logging.level="all:debug"), and you should see it fall-though the more specific choices before settling on the system symlink.

The weird thing about what you’re reporting tho is this: Error: failed to create ambient nodeagent service: error configuring iptables: binary iptables-save not found in path: exit status 127. This is very odd, as Istio’s distroless images should (or did, at least) have the local default symlinks of iptables and iptables-save in PATH.

Either way, I will open a PR to prefer nft unconditionally when we find both legacy and nft and the tiebreaker doesn’t tiebreak, which IMO we should do anyway.