istio: istio-cni : istio-iptables.sh no such file or directory

Bug description

When using istio-cni, the istio-validation container is unable to complete because the nsenter istio-iptables.sh command fails with no such file or directory.

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure [ ] Docs [ ] Installation [X] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Affected features (please put an X in all that apply)

[ ] Multi Cluster [ ] Virtual Machine [ ] Multi Control Plane

Expected behavior

following the istio-cni docs (https://istio.io/latest/docs/setup/additional-setup/cni/), I would expect that when I launch a container, istio-cni would properly setup the iptables rules for my new container.

Steps to reproduce the bug

  1. install istio (I’ve tried with 1.5.4. 1.6.0, 1.6.4 and 1.6.5)
  2. configure cni
  3. label the default namespace
  4. launch httpbin

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

client version: 1.6.3
control plane version: 1.6.3
data plane version: 1.6.3 (2 proxies)

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"09708a7b094a83f86b3ca5834a36376a36c9c564", GitTreeState:"clean", BuildDate:"2020-01-06T23:47:30Z", GoVersion:"go1.13.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?

$ istioctl install --set profile=default
$ cat <<EOF > istio-cni.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    cni:
      enabled: true
      namespace: kube-system
  values:
    cni:
      excludeNamespaces:
       - istio-system
       - kube-system
      logLevel: info
EOF
$ istioctl install -f istio-cni.yaml

Environment where bug was observed (cloud vendor, OS, etc)

aws

Additionally, please consider attaching a cluster state archive by attaching the dump file to this issue.

describe pod returns:

Events:
  Type     Reason                  Age              From                                               Message
  ----     ------                  ----             ----                                               -------
  Normal   Scheduled               7s               default-scheduler                                  Successfully assigned default/httpbin-654c6cbbb9-dt6q5 to ip-10-3-60-93.eu-west-1.compute.internal
  Warning  FailedCreatePodSandBox  6s               kubelet, ip-10-3-60-93.eu-west-1.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7d8646053f58dc26bbe2ef376dd43b28307cd47c1aa71eb42efd9a02090ed37f" network for pod "httpbin-654c6cbbb9-dt6q5": NetworkPlugin cni failed to set up pod "httpbin-654c6cbbb9-dt6q5_default" network: exit status 1
  Warning  FailedCreatePodSandBox  4s               kubelet, ip-10-3-60-93.eu-west-1.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a2a2914502bd8f45d0514fd288dcf4efe1eddaf10adc5d31eaa77c2e2d5a4057" network for pod "httpbin-654c6cbbb9-dt6q5": NetworkPlugin cni failed to set up pod "httpbin-654c6cbbb9-dt6q5_default" network: exit status 1
  Warning  FailedCreatePodSandBox  3s               kubelet, ip-10-3-60-93.eu-west-1.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2a9b41a535f6ce7922623f8be454949acfca15f5e52bd8abcdab3b07e74aee59" network for pod "httpbin-654c6cbbb9-dt6q5": NetworkPlugin cni failed to set up pod "httpbin-654c6cbbb9-dt6q5_default" network: exit status 1
  Warning  FailedCreatePodSandBox  2s               kubelet, ip-10-3-60-93.eu-west-1.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ed106c0c4d049851c9bf38ad68443fe29ba1ac4da46ac84ed87ceb8fc1590bb9" network for pod "httpbin-654c6cbbb9-dt6q5": NetworkPlugin cni failed to set up pod "httpbin-654c6cbbb9-dt6q5_default" network: exit status 1
  Warning  FailedCreatePodSandBox  1s               kubelet, ip-10-3-60-93.eu-west-1.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e84882e692285c4b0eb2a2eabd02d1b0d3e833691254c1589e30e51861e65548" network for pod "httpbin-654c6cbbb9-dt6q5": NetworkPlugin cni failed to set up pod "httpbin-654c6cbbb9-dt6q5_default" network: exit status 1
  Normal   SandboxChanged          0s (x6 over 5s)  kubelet, ip-10-3-60-93.eu-west-1.compute.internal  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  0s               kubelet, ip-10-3-60-93.eu-west-1.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b338e818279d491703af218002297588f784e312a0426503a8bef8b3800a13ca" network for pod "httpbin-654c6cbbb9-dt6q5": NetworkPlugin cni failed to set up pod "httpbin-654c6cbbb9-dt6q5_default" network: exit status 1

and the kubelet logs show

Jul 22 18:50:49 ip-10-3-60-93.eu-west-1.compute.internal kubelet-wrapper[1474]: {"level":"info","time":"2020-07-22T18:50:49.290103Z","msg":"setting up redirect"}
Jul 22 18:50:49 ip-10-3-60-93.eu-west-1.compute.internal kubelet-wrapper[1474]: {"level":"info","time":"2020-07-22T18:50:49.290113Z","msg":"Redirect local ports: *"}
Jul 22 18:50:49 ip-10-3-60-93.eu-west-1.compute.internal kubelet-wrapper[1474]: {"level":"info","time":"2020-07-22T18:50:49.290119Z","msg":"nsenter args","nsenterArgs":["--net=/proc/10355/ns/net","/opt/cni/bin/istio-iptables.sh","-p","15001","-u","1337","-m","REDIRECT","-i","*","-b","*","-d","15020,15021,15090","-o","15020","-x","","-k",""]}
Jul 22 18:50:49 ip-10-3-60-93.eu-west-1.compute.internal kubelet-wrapper[1474]: {"level":"error","time":"2020-07-22T18:50:49.291030Z","msg":"nsenter failed","out":"nsenter: failed to execute /opt/cni/bin/istio-iptables.sh: No such file or directory\n","error":"exit status 1"}
Jul 22 18:50:49 ip-10-3-60-93.eu-west-1.compute.internal kubelet-wrapper[1474]: {"level":"info","time":"2020-07-22T18:50:49.291049Z","msg":"nsenter out: nsenter: failed to execute /opt/cni/bin/istio-iptables.sh: No such file or directory\n"}
Jul 22 18:50:49 ip-10-3-60-93.eu-west-1.compute.internal kubelet-wrapper[1474]: E0722 18:50:49.291822    1474 cni.go:331] Error adding default_httpbin-654c6cbbb9-dt6q5/425ca3c8fad7e0e8468bc8b2063d8412eefc71df444c6d1c04d94ba505602f6c to network weave-net/weave: exit status 1
Jul 22 18:50:49 ip-10-3-60-93.eu-west-1.compute.internal kubelet-wrapper[1474]: {"level":"info","time":"2020-07-22T18:50:49.298847Z","msg":"istio-cni cmdDel parsing config"}
Jul 22 18:50:49 ip-10-3-60-93.eu-west-1.compute.internal kubelet-wrapper[1474]: E0722 18:50:49.467333    1474 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "425ca3c8fad7e0e8468bc8b2063d8412eefc71df444c6d1c04d94ba505602f6c" network for pod "httpbin-654c6cbbb9-dt6q5": NetworkPlugin cni failed to set up pod "httpbin-654c6cbbb9-dt6q5_default" network: exit status 1
Jul 22 18:50:49 ip-10-3-60-93.eu-west-1.compute.internal kubelet-wrapper[1474]: E0722 18:50:49.467394    1474 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "httpbin-654c6cbbb9-dt6q5_default(18cf035a-b482-4927-ae00-2a904a86f809)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "425ca3c8fad7e0e8468bc8b2063d8412eefc71df444c6d1c04d94ba505602f6c" network for pod "httpbin-654c6cbbb9-dt6q5": NetworkPlugin cni failed to set up pod "httpbin-654c6cbbb9-dt6q5_default" network: exit status 1
Jul 22 18:50:49 ip-10-3-60-93.eu-west-1.compute.internal kubelet-wrapper[1474]: E0722 18:50:49.467423    1474 kuberuntime_manager.go:692] createPodSandbox for pod "httpbin-654c6cbbb9-dt6q5_default(18cf035a-b482-4927-ae00-2a904a86f809)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "425ca3c8fad7e0e8468bc8b2063d8412eefc71df444c6d1c04d94ba505602f6c" network for pod "httpbin-654c6cbbb9-dt6q5": NetworkPlugin cni failed to set up pod "httpbin-654c6cbbb9-dt6q5_default" network: exit status 1

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 18 (9 by maintainers)

Most upvoted comments

So I followed the install steps used here in a Kind cluster and I’m not seeing any problems. I’m wondering if this may be an issue with AWS specifically.

@pmoody- Can you post details about the AWS cluster you are using? I’d like the following data points:

  • Is it EKS?
  • Are you using standard node images, or do you have a custom image?
  • What are the configuration and arguments passed to kubelet?
  • What CNI is it using?

At the moment I suspect that you may need an additional CNI argument to change the install location for the binaries and istio-iptables.sh script.

+1
stewartbutler on Jul 23, 2020 
$ k describe ds -n kube-system istio-cni-node | grep -B 1 Image
   install-cni:
    Image:      docker.io/istio/install-cni:1.6.3
--
   repair-cni:
    Image:      docker.io/istio/install-cni:1.6.3
+1
pmoody- on Jul 23, 2020
Read more comments on GitHub
← istio: Istio cannot be deployed on IPv6-first DS clusters
istio: istio-cni does not return an error when it cannot get pod information from k8s →