kubernetes: kube-proxy should log the payload when iptables-restore fails

my k8s cluster version is 1.17.x 。this cluster has 10 nodes.one master 。 only 2 nodes get problem.

E0809 05:37:21.484590       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 51 failed
)
I0809 05:37:21.484752       1 proxier.go:779] Sync failed; retrying in 30s
E0809 06:14:37.485082       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 53 failed
)
I0809 06:14:37.485205       1 proxier.go:779] Sync failed; retrying in 30s
E0809 06:45:24.676775       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 53 failed
)
I0809 06:45:24.676868       1 proxier.go:779] Sync failed; retrying in 30s
E0809 06:45:38.141258       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 52 failed
)
I0809 06:45:38.141408       1 proxier.go:779] Sync failed; retrying in 30s
E0809 07:32:42.616867       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 52 failed
)
I0809 07:32:42.617009       1 proxier.go:779] Sync failed; retrying in 30s
E0809 07:42:13.864568       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 51 failed
)
I0809 07:42:13.864698       1 proxier.go:779] Sync failed; retrying in 30s
E0809 08:00:20.842378       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 51 failed
)
I0809 08:00:20.842502       1 proxier.go:779] Sync failed; retrying in 30s
E0809 08:06:32.562513       1 proxier.go:1507] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 51 failed
)
I0809 08:06:32.562665       1 proxier.go:779] Sync failed; retrying in 30s

it make my svc sometime can use。sometime can timeout。 what can i do can resolve this problem ? this 2 node’s svc is nodeportType 。

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 22 (16 by maintainers)

Most upvoted comments

I’d start with pkg/util/iptables/iptables.go - restoreInternal()

It’s worth thinking about the cleanest solution here, but maybe something like:

type ParseError interface {
    Line() Int
}

type parseError struct {
    cmd string
    line int
}

func (e parseError) Line() int {
    return line
}

func (e parseError) Error() string {
    return fmt.Sprintf("%s: parse error on line %d", e.cmd, e.line)
} 

When you get an error from exec, parse the error for line %d failed and if it matches, return a parseError. Then let the callers check for a ParseError by interface assertion, extract the Line(), extract their input’s data at -3 and +3 from that line and log it. E.g.

 Failed to execute iptables-restore: input error on line 1234:
    1231: ...
    1232: ...
    1233: ...
    1234: ...
    1235: ...
    1236: ...
    1237: ...

You’ll proably want a helper func to do that, which could live in that same iptables package. And a test.

You’ll proably want a helper func to do that, which could live in that same iptables package. And a test.

you may need to write a helper function to extract them from the logs. my 2 cents… here is my version for reference. you could add some tests, to cover the boundary conditions.

const delimiter = 0x0a

func extratLines(limit, lineNumber int, lines []byte) string {
	lineBytes := bytes.Split(lines, []byte{delimiter})

	count := len(lineBytes)
	var extractedLines bytes.Buffer

	// validate boundary limits
	start := max(lineNumber-limit, 0)
	end := min(lineNumber+limit, count-1)

	for i := start; i <= end; i++ {
		extractedLines.WriteString(string(lineBytes[i]) + "\n")
	}

	return extractedLines.String()
}

func max(a, b int) int {
	if a > b {
		return a
	}

	return b
}

func min(a, b int) int {
	if a < b {
		return a
	}

	return b
}

to play around 😃 https://go.dev/play/p/STFKNN-4awR

Hi @cyclinder, apologies I was busy with some other work. You must please continue.

at least I didn’t see any PR from @amustaque97 , in case of collision please sync between both to avoid duplicate efforts