istio: panic in istio-init

For those who end up in this thread from the google search. I was experiencing the same problem, in my case the failing line number was on the COMMIT.

It turned out that SELinux was blocking the iptables change. Try to disable the SELinux temporarily by doing sudo setenforce 0 and then re-run your istioctl kube-inject... command to verify this is the issue.

@rlenglet Similar error in istio-init on GKE. 
GKE: v1.14.10-gke.24 
istio: 1.5.1 (Also I have tested on Istio 1.5.0 and I have got the same issue) NGINX Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1 # tells deployment to run 2 pods matching the template
  template: # create pods using pod definition in this template
    metadata:
      labels:
        app: nginx
        type: webserver
      annotations:
        sidecar.istio.io/interceptionMode: TPROXY
        sidecar.istio.io/proxyImage: docker.io/istio/proxyv2:1.5.0
    spec:
      containers:
      - name: nginx
        image: nginx:1.12.1
        ports:
        - containerPort: 80
          name: http-server

kubectl logs -f nginx-5964bfcd-9jxgd -c istio-init
Environment:
------------
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=TPROXY
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=80
INBOUND_PORTS_EXCLUDE=15090,15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBEVIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false

ip -f inet rule add fwmark 1337 lookup 133
ip -f inet route add local default dev lo table 133
RTNETLINK answers: File exists
ip route show table all
local default dev lo table 133 scope host
default via 10.24.0.1 dev eth0
10.24.0.0/24 via 10.24.0.1 dev eth0 src 10.24.0.12
10.24.0.1 dev eth0 scope link src 10.24.0.12
broadcast 10.24.0.0 dev eth0 table local proto kernel scope link src 10.24.0.12
local 10.24.0.12 dev eth0 table local proto kernel scope host src 10.24.0.12
broadcast 10.24.0.255 dev eth0 table local proto kernel scope link src 10.24.0.12
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
Writing following contents to rules file:  /tmp/iptables-rules-1585226543293516399.txt869048266
* mangle
-N ISTIO_DIVERT
-N ISTIO_TPROXY
-N ISTIO_INBOUND
-A ISTIO_DIVERT -j MARK --set-mark 1337
-A ISTIO_DIVERT -j ACCEPT
-A ISTIO_TPROXY ! -d 127.0.0.1/32 -p tcp -j TPROXY --tproxy-mark 1337/0xffffffff --on-port 15001
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 80 -m socket -j ISTIO_DIVERT
-A ISTIO_INBOUND -p tcp --dport 80 -m socket -j ISTIO_DIVERT
-A ISTIO_INBOUND -p tcp --dport 80 -j ISTIO_TPROXY
COMMIT
* nat
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port 15001
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT

iptables-restore --noflush /tmp/iptables-rules-1585226543293516399.txt869048266
iptables-restore: line 12 failed
iptables-save
# Generated by iptables-save v1.6.1 on Thu Mar 26 12:42:23 2020
*nat
:PREROUTING ACCEPT [43:2580]
:INPUT ACCEPT [43:2580]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Thu Mar 26 12:42:23 2020
# Generated by iptables-save v1.6.1 on Thu Mar 26 12:42:23 2020
*mangle
:PREROUTING ACCEPT [48:2906]
:INPUT ACCEPT [45:2700]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [45:1800]
:POSTROUTING ACCEPT [45:1800]
COMMIT
# Completed on Thu Mar 26 12:42:23 2020
panic: exit status 1

goroutine 1 [running]:
istio.io/istio/tools/istio-iptables/pkg/dependencies.(*RealDependencies).RunOrFail(0xd819c0, 0x9739b8, 0x10, 0xc000098ca0, 0x2, 0x2)
	istio.io/istio@/tools/istio-iptables/pkg/dependencies/implementation.go:44 +0x96
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeIptablesRestoreCommand(0xc000101d30, 0x7ff4c8630d01, 0x0, 0x0)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:474 +0x3aa
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeCommands(0xc000101d30)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:481 +0x45
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).run(0xc000101d30)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:428 +0x24e2
istio.io/istio/tools/istio-iptables/pkg/cmd.glob..func1(0xd5c740, 0xc0000e2700, 0x0, 0x10)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:56 +0x14e
github.com/spf13/cobra.(*Command).execute(0xd5c740, 0xc0000aa010, 0x10, 0x11, 0xd5c740, 0xc0000aa010)
	github.com/spf13/cobra@v0.0.5/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xd5c740, 0x40574f, 0xc00007e058, 0x0)
	github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v0.0.5/command.go:864
istio.io/istio/tools/istio-iptables/pkg/cmd.Execute()
	istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:284 +0x2d
main.main()
	istio.io/istio@/tools/istio-iptables/main.go:22 +0x20

We actually had the exact same issue in Istio/CNI: https://github.com/istio/istio/issues/15895, https://github.com/istio/cni/pull/172. We solved that by just ignoring the errors. But I think these rules are not really required in the first place. I will just remove them.

this is output from successful run

2019-12-13T21:38:18.370036467Z stdout F Environment:
2019-12-13T21:38:18.373057283Z stdout F ------------
2019-12-13T21:38:18.373187942Z stdout F ENVOY_PORT=
2019-12-13T21:38:18.373273953Z stdout F INBOUND_CAPTURE_PORT=
2019-12-13T21:38:18.373368583Z stdout F ISTIO_INBOUND_INTERCEPTION_MODE=
2019-12-13T21:38:18.373449073Z stdout F ISTIO_INBOUND_TPROXY_MARK=
2019-12-13T21:38:18.373524189Z stdout F ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
2019-12-13T21:38:18.373604077Z stdout F ISTIO_INBOUND_PORTS=
2019-12-13T21:38:18.373679734Z stdout F ISTIO_LOCAL_EXCLUDE_PORTS=
2019-12-13T21:38:18.373752203Z stdout F ISTIO_SERVICE_CIDR=
2019-12-13T21:38:18.373833484Z stdout F ISTIO_SERVICE_EXCLUDE_CIDR=
2019-12-13T21:38:18.373904908Z stdout F 
2019-12-13T21:38:18.373981985Z stdout F Variables:
2019-12-13T21:38:18.374228731Z stdout F ----------
2019-12-13T21:38:18.37434998Z stdout F PROXY_PORT=15001
2019-12-13T21:38:18.374439807Z stdout F PROXY_INBOUND_CAPTURE_PORT=15006
2019-12-13T21:38:18.374515118Z stdout F PROXY_UID=1337
2019-12-13T21:38:18.374597317Z stdout F INBOUND_INTERCEPTION_MODE=REDIRECT
2019-12-13T21:38:18.374678043Z stdout F INBOUND_TPROXY_MARK=1337
2019-12-13T21:38:18.374752795Z stdout F INBOUND_TPROXY_ROUTE_TABLE=133
2019-12-13T21:38:18.374826478Z stdout F INBOUND_PORTS_INCLUDE=*
2019-12-13T21:38:18.374940766Z stdout F INBOUND_PORTS_EXCLUDE=15020
2019-12-13T21:38:18.375111079Z stdout F OUTBOUND_IP_RANGES_INCLUDE=*
2019-12-13T21:38:18.375196003Z stdout F OUTBOUND_IP_RANGES_EXCLUDE=
2019-12-13T21:38:18.375279819Z stdout F OUTBOUND_PORTS_EXCLUDE=
2019-12-13T21:38:18.375364846Z stdout F KUBEVIRT_INTERFACES=
2019-12-13T21:38:18.375448374Z stdout F ENABLE_INBOUND_IPV6=
2019-12-13T21:38:18.375524588Z stdout F 
2019-12-13T21:38:18.377131534Z stdout F Writing following contents to rules file:  /tmp/iptables-rules-1576273098376111644.txt803754999
2019-12-13T21:38:18.377315014Z stdout F * nat
2019-12-13T21:38:18.377328938Z stdout F -N ISTIO_REDIRECT
2019-12-13T21:38:18.377334809Z stdout F -N ISTIO_IN_REDIRECT
2019-12-13T21:38:18.377340003Z stdout F -N ISTIO_INBOUND
2019-12-13T21:38:18.377345158Z stdout F -N ISTIO_OUTPUT
2019-12-13T21:38:18.377349873Z stdout F -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port 15001
2019-12-13T21:38:18.377356573Z stdout F -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port 15006
2019-12-13T21:38:18.377361267Z stdout F -A PREROUTING -p tcp -j ISTIO_INBOUND
2019-12-13T21:38:18.377377482Z stdout F -A ISTIO_INBOUND -p tcp --dport 22 -j RETURN
2019-12-13T21:38:18.377382658Z stdout F -A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
2019-12-13T21:38:18.377387584Z stdout F -A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
2019-12-13T21:38:18.377392223Z stdout F -A OUTPUT -p tcp -j ISTIO_OUTPUT
2019-12-13T21:38:18.377396795Z stdout F -A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
2019-12-13T21:38:18.377401654Z stdout F -A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -j ISTIO_IN_REDIRECT
2019-12-13T21:38:18.377406909Z stdout F -A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
2019-12-13T21:38:18.377411444Z stdout F -A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
2019-12-13T21:38:18.377415779Z stdout F -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
2019-12-13T21:38:18.377421684Z stdout F -A ISTIO_OUTPUT -j ISTIO_REDIRECT
2019-12-13T21:38:18.377426439Z stdout F COMMIT
2019-12-13T21:38:18.377430963Z stdout F 
2019-12-13T21:38:18.3776478Z stdout F iptables-restore --noflush /tmp/iptables-rules-1576273098376111644.txt803754999
2019-12-13T21:38:18.596511741Z stdout F Writing following contents to rules file:  /tmp/ip6tables-rules-1576273098596003250.txt249846506
2019-12-13T21:38:18.596663604Z stdout F * filter
2019-12-13T21:38:18.596674699Z stdout F -A INPUT -m state --state ESTABLISHED -j ACCEPT
2019-12-13T21:38:18.59668036Z stdout F -A INPUT -i lo -d ::1 -j ACCEPT
2019-12-13T21:38:18.596686183Z stdout F -A INPUT -j REJECT
2019-12-13T21:38:18.596692474Z stdout F COMMIT
2019-12-13T21:38:18.596696759Z stdout F 
2019-12-13T21:38:18.596811114Z stdout F ip6tables-restore --noflush /tmp/ip6tables-rules-1576273098596003250.txt249846506
2019-12-13T21:38:18.975058837Z stdout F iptables-save 
2019-12-13T21:38:18.975120125Z stdout F # Generated by iptables-save v1.6.1 on Fri Dec 13 21:38:18 2019
2019-12-13T21:38:18.975129896Z stdout F *nat
2019-12-13T21:38:18.975139265Z stdout F :PREROUTING ACCEPT [0:0]
2019-12-13T21:38:18.975147577Z stdout F :INPUT ACCEPT [0:0]
2019-12-13T21:38:18.975154667Z stdout F :OUTPUT ACCEPT [0:0]
2019-12-13T21:38:18.975161582Z stdout F :POSTROUTING ACCEPT [0:0]
2019-12-13T21:38:18.975170731Z stdout F :ISTIO_INBOUND - [0:0]
2019-12-13T21:38:18.975177029Z stdout F :ISTIO_IN_REDIRECT - [0:0]
2019-12-13T21:38:18.975183205Z stdout F :ISTIO_OUTPUT - [0:0]
2019-12-13T21:38:18.975189069Z stdout F :ISTIO_REDIRECT - [0:0]
2019-12-13T21:38:18.975195113Z stdout F -A PREROUTING -p tcp -j ISTIO_INBOUND
2019-12-13T21:38:18.97520316Z stdout F -A OUTPUT -p tcp -j ISTIO_OUTPUT
2019-12-13T21:38:18.975209434Z stdout F -A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
2019-12-13T21:38:18.975217294Z stdout F -A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
2019-12-13T21:38:18.975223339Z stdout F -A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
2019-12-13T21:38:18.975231251Z stdout F -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
2019-12-13T21:38:18.975237599Z stdout F -A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
2019-12-13T21:38:18.975243791Z stdout F -A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ISTIO_IN_REDIRECT
2019-12-13T21:38:18.975250074Z stdout F -A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
2019-12-13T21:38:18.975256626Z stdout F -A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
2019-12-13T21:38:18.97526343Z stdout F -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
2019-12-13T21:38:18.975269588Z stdout F -A ISTIO_OUTPUT -j ISTIO_REDIRECT
2019-12-13T21:38:18.975275916Z stdout F -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
2019-12-13T21:38:18.975281752Z stdout F COMMIT
2019-12-13T21:38:18.975288165Z stdout F # Completed on Fri Dec 13 21:38:18 2019
2019-12-13T21:38:18.978456844Z stdout F ip6tables-save 
2019-12-13T21:38:18.992488404Z stdout F # Generated by ip6tables-save v1.6.1 on Fri Dec 13 21:38:18 2019
2019-12-13T21:38:18.992522862Z stdout F *filter
2019-12-13T21:38:18.992530935Z stdout F :INPUT ACCEPT [0:0]
2019-12-13T21:38:18.992537739Z stdout F :FORWARD ACCEPT [0:0]
2019-12-13T21:38:18.99254371Z stdout F :OUTPUT ACCEPT [0:0]
2019-12-13T21:38:18.992549844Z stdout F -A INPUT -m state --state ESTABLISHED -j ACCEPT
2019-12-13T21:38:18.992556469Z stdout F -A INPUT -d ::1/128 -i lo -j ACCEPT
2019-12-13T21:38:18.992563475Z stdout F -A INPUT -j REJECT --reject-with icmp6-port-unreachable
2019-12-13T21:38:18.99257032Z stdout F COMMIT
2019-12-13T21:38:18.992576449Z stdout F # Completed on Fri Dec 13 21:38:18 2019

this should be in our init image right? Does not have anything to do with circleci… But something is failing…