istio: on mac m1 : error Command error output: xtables parameter problem: iptables-restore: unable to initialize table 'nat'

I do not understand the problem, and cannot reproduce this error by running iptables-restore with the rules in a file in a privileged container. The rules from manual invocation are confirmed within the nat table. The problem is not with the underlying Linux configuration.

I have tested with the containers built by the querycap/istio project and can confirm them to be fully functional on arm64 on my Apple M1 Max Macbook Pro in k3d on Docker for Mac. The problem appears to be with the packaging of these images.

As a workaround use the images provided by querycap/istio, either with their IstioOperator suggestion or by changing the instances of docker.io/istio to docker.io/querycapistio in the Istio manifests.

EDIT: querycap/istio is no longer keeping current. See my comment below for current recommendations.

This issue is still not resolved

This is still an issue.

@howardjohn My preliminary testing indicates that the development build 1.16-alpha.db637b3e4d3c09da0f8fc8f3222231e1834668ed does indeed resolve the issue.

The command I used to generate the manifests used for testing was:

istioctl manifest generate \
    --set values.global.hub=gcr.io/istio-testing \
    --set values.global.tag=1.16-alpha.db637b3e4d3c09da0f8fc8f3222231e1834668ed

I have the same problem and I hope istio can support mac apple silicon someday.😎

I can confirm that the querycap workaround pointed to by @oursland works in my Kind cluster running on Macbook M1.

Kind version: 0.12.0 Istio version: 1.13.2

This is my Istio config.yaml (ref from https://github.com/querycap/istio ):

kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  hub: docker.io/querycapistio
  profile: minimal

Installed Istio using: istioctl install -f config.yaml.

I am able to see this example work: kubectl exec "$(kubectl get pod -l app=sleep -n bar -o jsonpath={.items..metadata.name})" -c sleep -n bar -- curl http://httpbin.foo:8000/ip -s -o /dev/null -w "%{http_code}\n" 200

I’ve also tried the above in an AWS EC2 Arm64 VM based Kind cluster. Istio runs and sidecars come up successfully, but httpbin or helloworld container samples do not come up because they are amd64 images. I think the Docker installation on Mac M1 allows amd64 images to cross-run on arm64 architecture, which means that I did not have a problem running the sample apps on M1/Kind clusters.

Rip m1 macs… 😦

Thanks @oursland ! If anyone else has issues on builds labeled 1.16-alpha.XYZ or 1.15.0+, please open an issue. thanks!

Any pointers to what may be happening here? It doesn’t look like the init container fails to come up on an ARM based kind cluster, and it also doesn’t look like iptables is failing because of an arch issue, as the command basically runs but fails to initialize the ‘nat’ table.

I’ve also tried this with istio-cni enabled - the istio-validation container doesn’t initialize and keeps getting restarted. So I haven’t been able to collect logs from it.

After having the error stated in this issue, I started minikube with “–network-plugin=cni”, then installed istio with “components.cni.enabled=true”, but now I get the following error:

% kubectl -n xxxxxxxx logs $(kubectl get pods -n xxxxxxxx| grep postgres | head -1 | awk '{print $1}') -c istio-validation
2022-02-22T07:58:23.322838Z	info	in new validator: 10.244.120.73
2022-02-22T07:58:23.325008Z	info	Listening on 127.0.0.1:15001
2022-02-22T07:58:23.328190Z	info	Listening on 127.0.0.1:15006
2022-02-22T07:58:23.332416Z	error	Error ipv4 getsockopt: protocol not available
2022-02-22T07:58:23.332640Z	error	Error getting original dst: protocol not available
2022-02-22T07:58:24.333488Z	error	Error ipv4 getsockopt: protocol not available
2022-02-22T07:58:24.333571Z	error	Error getting original dst: protocol not available

Alright, there are two possibilities here I think

1. Due to no arm64 support in Istio, qemu is used and that doesn’t behave well with the iptables. This is now fixed! Someone should try https://github.com/istio/istio/issues/26652#issuecomment-1205368297 on an m1 mac and report back
2. If not ^, then it must be some kernel params in the docker desktop VM. I don’t know much about docker desktop, but we would need to make sure it meets https://istio.io/latest/docs/setup/platform-setup/prerequisites/#kernel-module-requirements-on-cluster-nodes

@scottleedavis @psloboda @myartemis @devutkarsh

querycapistio is not keeping up with the latest releases, but resf/istio is. You’ll find their images at the project’s GitHub Packages.

If you’re using Istio Operator, then their instructions on the README.md will be get you set up.

If you’re not (I use manifests), then point the hub and images to their image registry as outlined in this PR to the README.md.

2022-04-18T19:35:26.437072Z	info	Writing following contents to rules file: /tmp/iptables-rules-1650310526436903351.txt3442681366
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
2022-04-18T19:35:26.437100Z	info	Running command: iptables-restore --noflush /tmp/iptables-rules-1650310526436903351.txt3442681366
2022-04-18T19:35:26.438644Z	error	Command error output: xtables parameter problem: iptables-restore: unable to initialize table 'nat'

Error occurred at line: 1
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
2022-04-18T19:35:26.438721Z	error	Failed to execute: iptables-restore --noflush /tmp/iptables-rules-1650310526436903351.txt3442681366, exit status 2

https://github.com/istio/istio/issues/23009#issuecomment-980864913

modprobe br_netfilter ; modprobe nf_nat ; modprobe xt_REDIRECT ; modprobe xt_owner; modprobe iptable_nat; modprobe iptable_mangle; modprobe iptable_filter

solved my problem. (virtualbox-almalinux8.5 in macos)

Same issue here would love to know an answer.

Running Minikube with Docker driver on my local Mac M1.

❯ istioctl version
client version: 1.13.1
control plane version: 1.13.1
data plane version: 1.13.1 (2 proxies)
❯ minikube version
minikube version: v1.25.2
commit: 362d5fdc0a3dbee389b3d3f1034e8023e72bd3a7

Issue is met after installing Istio on my cluster with istioctl install and deploying my pods giving Init:CrashLoopBackOff status and the similar error with iptables:

❯ kubectl logs photo-service-6b48d95b49-xbk9c -c istio-init
2022-03-02T17:44:09.756562Z	info	Istio iptables environment:
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_EXCLUDE_INTERFACES=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
ISTIO_META_DNS_CAPTURE=
INVALID_DROP=

2022-03-02T17:44:09.758734Z	info	Istio iptables variables:
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15021,15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBE_VIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false
DNS_CAPTURE=false
DROP_INVALID=false
CAPTURE_ALL_DNS=false
DNS_SERVERS=[],[]
OUTPUT_PATH=
NETWORK_NAMESPACE=
CNI_MODE=false
EXCLUDE_INTERFACES=

2022-03-02T17:44:09.760867Z	info	Writing following contents to rules file: /tmp/iptables-rules-1646243049760403841.txt3610950387
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
2022-03-02T17:44:09.761113Z	info	Running command: iptables-restore --noflush /tmp/iptables-rules-1646243049760403841.txt3610950387
2022-03-02T17:44:09.783539Z	error	Command error output: xtables parameter problem: iptables-restore: unable to initialize table 'nat'

Error occurred at line: 1
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
2022-03-02T17:44:09.783854Z	error	Failed to execute: iptables-restore --noflush /tmp/iptables-rules-1646243049760403841.txt3610950387, exit status 2

Its in 1.15.0 - just wasn’t in the original 1.15 betas which is why we had 1.16 discussion above

Any pointers to what may be happening here? It doesn’t look like the init container fails to come up on an ARM based kind cluster, and it also doesn’t look like iptables is failing because of an arch issue, as the command basically runs but fails to initialize the ‘nat’ table.

I’ve also tried this with istio-cni enabled - the istio-validation container doesn’t initialize and keeps getting restarted. So I haven’t been able to collect logs from it.

Seeing the same issue as well on M1 Macbook Pro. Kubernetes provider: Kind v0.11.1 Kind Docker image kernel: 5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

Istiod comes up in istio-system namespace. Istio version: 1.12.1

However sleep pod does not come up in a namespace marked with istio-injection=enabled, and fails at istio-init container with the same error reported above: iptables-restore v1.6.1: iptables-restore: unable to initialize table ‘nat’

Error occurred at line: 1 Try `iptables-restore -h’ or ‘iptables-restore --help’ for more information. iptables-save panic: exit status 2

IPtables modules seem to be installed in the Kind image kernel: `modprobe br_netfilter --first-time; modprobe nf_nat --first-time; modprobe xt_REDIRECT --first-time; modprobe xt_owner --first-time; modprobe iptable_nat --first-time; modprobe iptable_mangle --first-time; modprobe iptable_filter --first-time

modprobe: ERROR: could not insert ‘br_netfilter’: Module already in kernel modprobe: ERROR: could not insert ‘nf_nat’: Module already in kernel modprobe: ERROR: could not insert ‘xt_REDIRECT’: Module already in kernel modprobe: ERROR: could not insert ‘xt_owner’: Module already in kernel modprobe: ERROR: could not insert ‘iptable_nat’: Module already in kernel modprobe: ERROR: could not insert ‘iptable_mangle’: Module already in kernel modprobe: ERROR: could not insert ‘iptable_filter’: Module already in kernel`