kubernetes: [sig-node] Summary API [NodeConformance] when querying /stats/summary ... networking info is nil on containerd

https://github.com/google/cadvisor/pull/3103 is ready now. the CI jobs for cadvisor are 🟢 now as well.

@bobbypage can you please merge it, release v0.45 and open a PR for k/k to update the version of k8s.

thanks, Dims

I don’t see the metrics grabber related to this, there is a bug on the getNodeSummary though, it doesn’t work for ipv6 addresses

diff --git a/test/e2e_node/util.go b/test/e2e_node/util.go
index 1c8b1699f74..8f803460e68 100644
--- a/test/e2e_node/util.go
+++ b/test/e2e_node/util.go
@@ -23,9 +23,11 @@ import (
        "flag"
        "fmt"
        "io"
+       "net"
        "net/http"
        "os/exec"
        "regexp"
+       "strconv"
        "strings"
        "time"
 
@@ -82,7 +84,7 @@ func getNodeSummary() (*stats.Summary, error) {
        if err != nil {
                return nil, fmt.Errorf("failed to get current kubelet config")
        }
-       req, err := http.NewRequest("GET", fmt.Sprintf("http://%s:%d/stats/summary", kubeletConfig.Address, kubeletConfig.ReadOnlyPort), nil)
+       req, err := http.NewRequest("GET", fmt.Sprintf("http://%s/stats/summary", net.JoinHostPort(kubeletConfig.Address, strconv.Itoa(kubeletConfig.ReadOnlyPort))), nil)
        if err != nil {
                return nil, fmt.Errorf("failed to build http request: %v", err)
        }

Also, the

			gomega.Eventually(getNodeSummary, 180*time.Second, 15*time.Second).Should(matchExpectations)
			ginkgo.By("Validating /stats/summary are consistent")
			// Then the summary should match the expectations a few more times.
			gomega.Consistently(getNodeSummary, 30*time.Second, 15*time.Second).Should(matchExpectations)

are not providing any information, I think we should use something different to compare to avoid artifacts failing the comparison.

I’ll submit a patch

btw, now we are also trying to repro bug with logging: https://github.com/kubernetes/kubernetes/pull/109472 so far no luck. @mmiranda96 and @bobbypage tried locally, never succeeded to repro. @ruiwen-zhao had some luck with repro before: https://github.com/kubernetes/kubernetes/pull/109371 but it seems not any longer.

@liggitt this test is generally very flaky in part because it tests so many (too many?) things…

https://github.com/kubernetes/kubernetes/issues/108836 had issues with CPUStats https://github.com/kubernetes/kubernetes/issues/104292 was swap-only?

This failure is networkstats timing out specifically.

reproduced with a bit more logging in https://github.com/kubernetes/kubernetes/pull/109371#issuecomment-1104422428

TLDR, my change below caused a “invalid memory address or nil pointer dereference” intermittently:

framework.Logf("ruiwen-zhao: /stats/suammry: %+v", summary)
	for _, p := range summary.Pods {
		framework.Logf("ruiwen-zhao: NetworkStats: %+v", *p.Network)
	}

So I guess p.Network is an empty pointer when the test fails.

@mikebrow we may try the approach like #101960 to fix this.

Also checking time of the test run in the latest 5 runs, and the earliest 5 runs, I don’t see a huge difference

// Latest 5 runs - all passing
E2eNode Suite: [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api	1m1s

E2eNode Suite: [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api	1m4s

E2eNode Suite: [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api	49s

E2eNode Suite: [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api	1m4s

E2eNode Suite: [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api	1m8s

---
// 5 runs after 2022/03/23 21:14:10, all passing

E2eNode Suite: [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api	55s

E2eNode Suite: [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api	1m6s

E2eNode Suite: [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api	50s

E2eNode Suite: [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api	1m4s

E2eNode Suite: [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api	51s

So the passing tests are pretty stable and take only ~ 1min, whereas failed runs take more than 3 mins. This does not look like a performance regression.

@helayoty this is one of three issues SIG Node is tracking as a possible regression in 1.24, we need a clear answer before it can be removed from the milestone.

👍 for explanation; 👎 for unclear test signal 😃