kubernetes: iptables-restore invalid option in kube-proxy

I have exactly same issue with kubernetes 1.11.1 and ipvs:

I0905 20:09:11.677674       1 iptables.go:323] running iptables-save [-t nat]
I0905 20:09:11.731479       1 iptables.go:383] running iptables-restore [-T nat --noflush --counters]
E0905 20:09:11.733419       1 proxier.go:423] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
)
I0905 20:09:11.733681       1 iptables.go:323] running iptables-save [-t filter]
I0905 20:09:11.737057       1 iptables.go:383] running iptables-restore [-T filter --noflush --counters]
I0905 20:09:11.738971       1 server.go:448] Version: v1.11.1

The log is only visible during kube-proxy startup (only)!

But there is a one strange thing. I am pretty sure that I am using ipvs:

I0905 20:09:11.567087       1 server_others.go:183] Using ipvs Proxier.
I0905 20:09:11.646988       1 proxier.go:346] nodeIP: 10.10.0.12, isIPv6: false
I0905 20:09:11.647037       1 proxier.go:403] minSyncPeriod: 0s, syncPeriod: 30s, burstSyncs: 2

But based on the log message:

proxier.go:423] Failed to execute iptables-restore for nat:

the logs is somehow from iptables proxier: https://github.com/kubernetes/kubernetes/blob/v1.11.1/pkg/proxy/iptables/proxier.go#L423

The ipvs proxier have similar output but different line: https://github.com/kubernetes/kubernetes/blob/v1.11.1/pkg/proxy/ipvs/proxier.go#L1139

How come that iptables proxier is executed when ipvs is configured?

For anyone finding this thread the below workaround may work for now with Centos at least:

Ok so as a work around I have done:

yum downgrade http://mirror.centos.org/centos/7.5.1804/updates/x86_64/Packages/iptables-1.4.21-24.1.el7_5.x86_64.rpm

and adding iptables-* to the exclude in /etc/yum.conf

[main]
.....
....
exclude=iptables-*

This is NOT a good fix as it rolling iptables back 4 releases but until kube-proxy is fixed this may be the only viable option for some

I don’t have experience in this area, but if you run somthing like:

# as root
strace -p ${KUBELET_PID} -s 8096 -f 2>&1 | tee /tmp/strace.log

you’ll be able to see the kubelet spawn the iptables-restore process, and see the input that it writes to it. If it’s writing directly to iptables-restore you might be able to limit the output by running:

strace -p ${KUBELET_PID} -e trace=process,write -s 8096 -f 2>&1 | tee /tmp/strace.log

This advise is more generally about debugging the invocation of processes, and is here only to provide some guide to debug this further. See man strace for details.

(the logs will be in /tmp/strace.log)

From journalctl:

Feb 20 18:45:57 sw-20160601-01 localkube[31719]: E0220 18:45:57.956015   31719 proxier.go:1667] Failed to execute iptables-restore: exit status 1 (iptables-restore: invalid option -- '5'
Feb 20 18:45:57 sw-20160601-01 localkube[31719]: Try `iptables-restore -h' for more information.
Feb 20 18:45:57 sw-20160601-01 localkube[31719]: )

From strace -p ${PID} -f -s 8096 -e process:

[pid 13285] execve("/sbin/iptables-restore", ["iptables-restore", "-w5", "--noflush", "--counters"], 0xc42eaecc80 /* 6 vars */ <unfinished ...>
[pid  5492] <... clone resumed> child_stack=NULL, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 13285
[pid 13285] <... execve resumed> )      = 0
[pid  5492] waitid(P_PID, 13285,  <unfinished ...>
[pid 13285] arch_prctl(ARCH_SET_FS, 0x7f63b37e0440) = 0
[pid 13285] exit_group(1)               = ?
[pid 13285] +++ exited with 1 +++

To me, this seems pretty definitive – that block is wrong.

$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux buster/sid"
NAME="Debian GNU/Linux"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ minikube version
minikube version: v0.25.0

$ sudo iptables --version
iptables v1.6.2

PR to fix:

https://github.com/kubernetes/kubernetes/pull/59181

But it’s not complete. There seem to be issues more generally with that chunk.