kubernetes: kube-proxy should log the payload when iptables-restore fails
my k8s cluster version is 1.17.x 。this cluster has 10 nodes.one master 。 only 2 nodes get problem.
E0809 05:37:21.484590 1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 51 failed
)
I0809 05:37:21.484752 1 proxier.go:779] Sync failed; retrying in 30s
E0809 06:14:37.485082 1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 53 failed
)
I0809 06:14:37.485205 1 proxier.go:779] Sync failed; retrying in 30s
E0809 06:45:24.676775 1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 53 failed
)
I0809 06:45:24.676868 1 proxier.go:779] Sync failed; retrying in 30s
E0809 06:45:38.141258 1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 52 failed
)
I0809 06:45:38.141408 1 proxier.go:779] Sync failed; retrying in 30s
E0809 07:32:42.616867 1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 52 failed
)
I0809 07:32:42.617009 1 proxier.go:779] Sync failed; retrying in 30s
E0809 07:42:13.864568 1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 51 failed
)
I0809 07:42:13.864698 1 proxier.go:779] Sync failed; retrying in 30s
E0809 08:00:20.842378 1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 51 failed
)
I0809 08:00:20.842502 1 proxier.go:779] Sync failed; retrying in 30s
E0809 08:06:32.562513 1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 51 failed
)
I0809 08:06:32.562665 1 proxier.go:779] Sync failed; retrying in 30s
it make my svc sometime can use。sometime can timeout。 what can i do can resolve this problem ? this 2 node’s svc is nodeportType 。
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 22 (16 by maintainers)
I’d start with pkg/util/iptables/iptables.go -
restoreInternal()
It’s worth thinking about the cleanest solution here, but maybe something like:
When you get an error from exec, parse the error for
line %d failed
and if it matches, return aparseError
. Then let the callers check for aParseError
by interface assertion, extract the Line(), extract their input’s data at -3 and +3 from that line and log it. E.g.You’ll proably want a helper func to do that, which could live in that same iptables package. And a test.
you may need to write a helper function to extract them from the logs. my 2 cents… here is my version for reference. you could add some tests, to cover the boundary conditions.
to play around 😃 https://go.dev/play/p/STFKNN-4awR
Hi @cyclinder, apologies I was busy with some other work. You must please continue.
at least I didn’t see any PR from @amustaque97 , in case of collision please sync between both to avoid duplicate efforts