istio: Ubuntu 24.04 + Vanilla k8s + Calico + 1.23 Ambient - istio-cni failing to start on 2/3 nodes
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
Node are set up 100% identical save for a couple minor hardware differences on one, but two are 100% identical and one is working and one is failing.
Installed “vanilla k8s” with kubeadm, installed calico as per docs, cluster is running fine. Following the Ambient install guide and failing at the second step, installing istio-cni from helm (though I’m having the same issue with istioctl).
Two of the pods, on the same two nodes every time, are dying and staying in a backoff loop with -
Error: failed to create ambient nodeagent service: error configuring iptables: binary iptables-save not found in path: exit status 127
Yet they’re using the exact same image as the node it is starting successfully on, and the systems are configured the same and have iptables installed and iptables-save available (not sure if it’s referring to within a container or on the host tbh).
Troubleshooting for a full day now, and all I can come up with is ‘bug’
Version
istioctl version -
Istio is not present in the cluster: no running Istio pods in namespace "istio-system"
client version: 1.23.0
kubectl version -
Client Version: v1.30.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.3
helm version --short -
v3.15.4+gfa9efb0
Additional Information
Target cluster context: vcluster
Running with the following config:
istio-namespace: istio-system full-secrets: false timeout (mins): 30 include: { } exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage } end-time: 2024-08-21 21:37:36.725950598 -0400 EDT
Cluster endpoint: https://vcluster-vip.mydomain.tld:6444 CLI version: version.BuildInfo{Version:“1.23.0”, GitRevision:“d0ca037e58f90f58b38b33c5769fec25b6bfa9f3”, GolangVersion:“go1.22.6”, BuildStatus:“Clean”, GitTag:“1.23.0”}
The following Istio control plane revisions/versions were found in the cluster: The following proxy revisions/versions were found in the cluster:
Fetching logs for the following containers:
calico-apiserver/calico-apiserver/calico-apiserver-85b898779c-779t4/calico-apiserver calico-apiserver/calico-apiserver/calico-apiserver-85b898779c-tdwxh/calico-apiserver calico-system/calico-kube-controllers/calico-kube-controllers-77b9c68f4-96gjf/calico-kube-controllers calico-system/calico-node/calico-node-26pk8/calico-node calico-system/calico-node/calico-node-4rz7x/calico-node calico-system/calico-node/calico-node-g665r/calico-node calico-system/calico-typha/calico-typha-7854c6d879-2xdgf/calico-typha calico-system/calico-typha/calico-typha-7854c6d879-mq872/calico-typha calico-system/csi-node-driver/csi-node-driver-4fcwh/calico-csi calico-system/csi-node-driver/csi-node-driver-4fcwh/csi-node-driver-registrar calico-system/csi-node-driver/csi-node-driver-rslkr/calico-csi calico-system/csi-node-driver/csi-node-driver-rslkr/csi-node-driver-registrar calico-system/csi-node-driver/csi-node-driver-v8hsn/calico-csi calico-system/csi-node-driver/csi-node-driver-v8hsn/csi-node-driver-registrar istio-system/istio-cni-node/istio-cni-node-dw5wb/install-cni istio-system/istio-cni-node/istio-cni-node-jqqgj/install-cni istio-system/istio-cni-node/istio-cni-node-r8wvj/install-cni tigera-operator/tigera-operator/tigera-operator-77f994b5bb-mhv54/tigera-operator
Fetching Istio control plane information from cluster.
Fetching CNI logs from cluster.
Running Istio analyze on all namespaces and report as below: Analysis Report: Info [IST0102] (Namespace calico-apiserver) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace calico-apiserver istio-injection=enabled’ to enable it, or ‘kubectl label namespace calico-apiserver istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace calico-system) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace calico-system istio-injection=enabled’ to enable it, or ‘kubectl label namespace calico-system istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace default istio-injection=enabled’ to enable it, or ‘kubectl label namespace default istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kube-node-lease) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kube-node-lease istio-injection=enabled’ to enable it, or ‘kubectl label namespace kube-node-lease istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kube-public) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kube-public istio-injection=enabled’ to enable it, or ‘kubectl label namespace kube-public istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kube-system) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kube-system istio-injection=enabled’ to enable it, or ‘kubectl label namespace kube-system istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace tigera-operator) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace tigera-operator istio-injection=enabled’ to enable it, or ‘kubectl label namespace tigera-operator istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0118] (Service calico-apiserver/calico-api) Port name apiserver (port: 443, targetPort: 5443) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service calico-system/calico-kube-controllers-metrics) Port name metrics-port (port: 9094, targetPort: 9094) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service calico-system/calico-typha) Port name calico-typha (port: 5473, targetPort: calico-typha) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service kube-system/kube-dns) Port name metrics (port: 9153, targetPort: 9153) doesn’t follow the naming convention of Istio port. Creating an archive at /home/me/bug-report.tar.gz. Time used for creating the tar file is 143.414681ms. Cleaning up temporary files in /tmp/bug-report. Done.
Please note the cluster is named vcluster, because I intend to install vcluster on it at some point, but it is not yet installed, and this is not a vcluster, it’s vanilla k8s running on Ubuntu VMs running on Proxmox.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 20 (9 by maintainers)
I’ve got the same issue with Istio version
1.23.2, deployed in a Lima VM (KVM) with Kind cluster.Deploying Istio with
variant: debugworks.At the top, you can see that it is a KVM. On the left, installing with
variant: distroless. On the right, the error log in theistio-cni-nodepodThis does not work:
this works:
Ah!
During the last round of failures I was running Ubuntu Server 22.04, I can spin up a VM later to check what the kernel/glib versions are on that.
Earlier attempts and what I’ve since reverted to were Ubuntu Server 24.04 minimal -
Our
istiobase image ships both variants, and for ambient we are only running from containers, so we attempt to select the correct in-container binary based on whether the host node’s (nft|legacy) tables already have rules in them.if we can’t “tiebreak” between
nft/legacyby checking rulecount, we always fall back to the local default symlink e…giptables/iptables-save).My suspicion is that on some of your nodes, you have
nftrules on the node already when we come up, which is whyistio-cni-node-r8wvjfor instance picksnft, and doesn’t have a problem.The other nodes probably don’t have any rules in either legacy or nft tables, at the time we check, so we fall back to the local default symlink.
(in this scenario it would be better to just prefer
nft, which would mask your problem here, as mentioned over here: https://github.com/istio/istio/issues/52357#issuecomment-2253300746)You could confirm this theory (if you wish, no worries if not, already a great bug report) by installing
istio-cniwith debug logging enabled (--set global.logging.level="all:debug"), and you should see it fall-though the more specific choices before settling on the system symlink.The weird thing about what you’re reporting tho is this:
Error: failed to create ambient nodeagent service: error configuring iptables: binary iptables-save not found in path: exit status 127. This is very odd, as Istio’sdistrolessimages should (or did, at least) have the local default symlinks ofiptablesandiptables-savein PATH.Either way, I will open a PR to prefer
nftunconditionally when we find bothlegacyandnftand thetiebreakerdoesn’t tiebreak, which IMO we should do anyway.cc @bleggett