kubernetes: iptables-restore: line 7 failed in kube-proxy

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/sig network

What happened:

Kube-proxy (k8s.gcr.io/kube-proxy-amd64:v1.11.1) deployed as daemonset with ipvs mode produce during startup following error:

E0905 20:09:11.733419       1 proxier.go:423] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
)

With verbosity -v5 the logs looks like this:

I0905 20:09:11.677674       1 iptables.go:323] running iptables-save [-t nat]
I0905 20:09:11.731479       1 iptables.go:383] running iptables-restore [-T nat --noflush --counters]
E0905 20:09:11.733419       1 proxier.go:423] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
)
I0905 20:09:11.733681       1 iptables.go:323] running iptables-save [-t filter]
I0905 20:09:11.737057       1 iptables.go:383] running iptables-restore [-T filter --noflush --counters]
I0905 20:09:11.738971       1 server.go:448] Version: v1.11.1

Looks like proxier is trying to perform cleanup part: https://github.com/kubernetes/kubernetes/blob/v1.11.1/pkg/proxy/iptables/proxier.go#L423 but failed. Such error is generated only once during startup.

What you expected to happen:

Error should not exist 😃

Environment:

Kubernetes version (use kubectl version): 1.11.1
Cloud provider or hardware configuration: none
OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS
Kernel (e.g. uname -a):
Install tools: custom script based on docker
Others:

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 2
Comments: 20 (2 by maintainers)

Most upvoted comments

same here on v1.13.2, throws this error when kube-proxy initializes.

Running on CoreOS in ipvs mode:

I0119 17:06:52.872279       1 server_others.go:189] Using ipvs Proxier.
I0119 17:06:52.872998       1 graceful_termination.go:160] Trying to delete rs: 10.233.0.3:9153/TCP/10.233.114.193:9153
I0119 17:06:52.873067       1 graceful_termination.go:174] Deleting rs: 10.233.0.3:9153/TCP/10.233.114.193:9153
I0119 17:06:52.873114       1 graceful_termination.go:160] Trying to delete rs: 10.233.0.3:9153/TCP/10.233.105.65:9153
I0119 17:06:52.873186       1 graceful_termination.go:174] Deleting rs: 10.233.0.3:9153/TCP/10.233.105.65:9153
I0119 17:06:52.873298       1 graceful_termination.go:160] Trying to delete rs: 10.233.0.1:443/TCP/10.230.142.97:443
I0119 17:06:52.873402       1 graceful_termination.go:174] Deleting rs: 10.233.0.1:443/TCP/10.230.142.97:443
I0119 17:06:52.873471       1 graceful_termination.go:160] Trying to delete rs: 10.233.0.1:443/TCP/10.230.142.115:443
I0119 17:06:52.873546       1 graceful_termination.go:174] Deleting rs: 10.233.0.1:443/TCP/10.230.142.115:443
I0119 17:06:52.873617       1 graceful_termination.go:160] Trying to delete rs: 10.233.0.1:443/TCP/10.230.142.106:443
I0119 17:06:52.873687       1 graceful_termination.go:171] Not deleting, RS 10.233.0.1:443/TCP/10.230.142.106:443: 1 ActiveConn, 0 InactiveConn
I0119 17:06:52.873788       1 graceful_termination.go:160] Trying to delete rs: 10.233.47.95:44134/TCP/10.233.97.65:44134
I0119 17:06:52.873869       1 graceful_termination.go:174] Deleting rs: 10.233.47.95:44134/TCP/10.233.97.65:44134
I0119 17:06:52.873953       1 graceful_termination.go:160] Trying to delete rs: 10.233.34.203:443/TCP/10.233.105.66:443
I0119 17:06:52.874024       1 graceful_termination.go:171] Not deleting, RS 10.233.34.203:443/TCP/10.233.105.66:443: 2 ActiveConn, 0 InactiveConn
I0119 17:06:52.874116       1 graceful_termination.go:160] Trying to delete rs: 10.233.63.6:80/TCP/10.233.107.194:9090
I0119 17:06:52.874190       1 graceful_termination.go:174] Deleting rs: 10.233.63.6:80/TCP/10.233.107.194:9090
I0119 17:06:52.874274       1 graceful_termination.go:160] Trying to delete rs: 10.233.0.3:53/TCP/10.233.114.193:53
I0119 17:06:52.874351       1 graceful_termination.go:174] Deleting rs: 10.233.0.3:53/TCP/10.233.114.193:53
I0119 17:06:52.874419       1 graceful_termination.go:160] Trying to delete rs: 10.233.0.3:53/TCP/10.233.105.65:53
I0119 17:06:52.874502       1 graceful_termination.go:174] Deleting rs: 10.233.0.3:53/TCP/10.233.105.65:53
I0119 17:06:52.874592       1 graceful_termination.go:160] Trying to delete rs: 10.233.0.3:53/UDP/10.233.114.193:53
I0119 17:06:52.874791       1 graceful_termination.go:174] Deleting rs: 10.233.0.3:53/UDP/10.233.114.193:53
I0119 17:06:52.874857       1 graceful_termination.go:160] Trying to delete rs: 10.233.0.3:53/UDP/10.233.105.65:53
I0119 17:06:52.874952       1 graceful_termination.go:174] Deleting rs: 10.233.0.3:53/UDP/10.233.105.65:53
I0119 17:06:52.875042       1 server_others.go:216] Tearing down inactive rules.
I0119 17:06:52.875147       1 graceful_termination.go:160] Trying to delete rs: 10.233.0.1:443/TCP/10.230.142.106:443
I0119 17:06:52.875245       1 graceful_termination.go:171] Not deleting, RS 10.233.0.1:443/TCP/10.230.142.106:443: 1 ActiveConn, 0 InactiveConn
I0119 17:06:52.875258       1 graceful_termination.go:160] Trying to delete rs: 10.233.34.203:443/TCP/10.233.105.66:443
I0119 17:06:52.875351       1 graceful_termination.go:171] Not deleting, RS 10.233.34.203:443/TCP/10.233.105.66:443: 2 ActiveConn, 0 InactiveConn
E0119 17:06:52.905624       1 proxier.go:432] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
)
I0119 17:06:52.910333       1 server.go:464] Version: v1.13.2
I0119 17:06:52.918208       1 config.go:102] Starting endpoints config controller
I0119 17:06:52.918223       1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I0119 17:06:52.918214       1 config.go:202] Starting service config controller
I0119 17:06:52.918309       1 controller_utils.go:1027] Waiting for caches to sync for service config controller
I0119 17:06:53.018645       1 controller_utils.go:1034] Caches are synced for service config controller
I0119 17:06:53.019004       1 controller_utils.go:1034] Caches are synced for endpoints config controller

dannyk81 on Jan 19, 2019

i update kube-proxy 1.14.2, the error message is disappear

Lincoln-dac on May 30, 2019

same problem 1.13.2 ,how fix it?

Lincoln-dac on May 30, 2019