kubernetes: [Flaky Test] In-tree Volumes: Error getting c-m metrics : the server could not find the requested resource

Which jobs are flaking:

pull-kubernetes-e2e-gce
ci-kubernetes-e2e-gci-gce

Which test(s) are flaking:

[sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly]
[sig-storage] In-tree Volumes [Driver: emptydir] [Testpattern: Inline-volume (default fs)] subPath should support file as subpath [LinuxOnly]
[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-link] [Testpattern: Pre-provisioned PV (default fs)] volumes should allow exec of files on the volume
[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing single file [LinuxOnly]
[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-bindmounted] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data
[sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Inline-volume (default fs)] volumes should store data
[sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Inline-volume (default fs)] subPath should be able to unmount after the subpath directory is deleted
[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should be able to unmount after the subpath directory is deleted
[sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory
[sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Inline-volume (default fs)] subPath should support non-existent path
[sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support existing directories when readOnly specified in the volumeSource
[sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should be able to unmount after the subpath directory is deleted
[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource
[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-link-bindmounted] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing single file [LinuxOnly]
[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (ext3)] volumes should allow exec of files on the volume
[sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works
[sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (ext3)] volumes should store data

Testgrid link:

https://testgrid.k8s.io/sig-release-master-informing#gce-cos-master-default&width=5&include-filter-by-regex=In-tree&sort-by-flakiness= https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-e2e-gce&include-filter-by-regex=In-tree&width=5&sort-by-flakiness=

Reason for failure: Failure fetching metrics:

Dec 15 22:01:17.942: Error getting c-m metrics : the server rejected our request for an unknown reason (get pods kube-controller-manager-e2e-4fc8c5bbfe-674b9-master:10252)
...
Causes: [
                    {
                        Type: "UnexpectedServerResponse",
                        Message: "unknown",
                        Field: "",
                    },
                ],

Dec  4 20:06:05.021: Error getting Kubelet e2e-f19735d0af-674b9-minion-group-l4ln metrics: the server is currently unable to handle the request (get nodes e2e-f19735d0af-674b9-minion-group-l4ln:10250)
...
Causes: [
                    {
                        Type: "UnexpectedServerResponse",
                        Message: "Error trying to reach service: 'dial tcp 10.40.0.4:10250: i/o timeout'",
                        Field: "",
                    },
                ],

Anything else we need to know: https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&text=Error getting c-m metrics&job=pull-kubernetes-e2e-gce%24&test=sig-storage

This appears to have restarted abruptly around 2/11-2/12

/sig storage /priority important-soon

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 24 (24 by maintainers)

Most upvoted comments

Ah, so first we should wait for the pod to exist, then be ready, then return metrics

liggitt on Mar 14, 2020

still happening:

Mar 12 18:52:19.595: Error getting c-m metrics : error waiting for controller manager pod to be running: %!w(<nil>)
Unexpected error:
    <*errors.errorString | 0xc000fd4150>: {
        s: "error waiting for controller manager pod to be running: %!w(<nil>)",
    }
    error waiting for controller manager pod to be running: %!w(<nil>)

message logging missed using the correct errors, so the messages aren’t helpful:

--- a/test/e2e/framework/metrics/metrics_grabber.go
+++ b/test/e2e/framework/metrics/metrics_grabber.go
@@ -170,7 +170,7 @@ func (g *Grabber) GrabFromControllerManager() (ControllerManagerMetrics, error)
        podName := fmt.Sprintf("%v-%v", "kube-controller-manager", g.masterName)
        g.waitForControllerManagerReadyOnce.Do(func() {
                if runningErr := e2epod.WaitForPodNameRunningInNamespace(g.client, podName, metav1.NamespaceSystem); runningErr != nil {
-                       err = fmt.Errorf("error waiting for controller manager pod to be running: %w", err)
+                       err = fmt.Errorf("error waiting for controller manager pod to be running: %w", runningErr)
                        return
                }

@@ -179,7 +179,7 @@ func (g *Grabber) GrabFromControllerManager() (ControllerManagerMetrics, error)
                        _, lastMetricsFetchErr = g.getMetricsFromPod(g.client, podName, metav1.NamespaceSystem, ports.InsecureKubeControllerManagerPort)
                        return lastMetricsFetchErr == nil, nil
                }); metricsWaitErr != nil {
-                       err = fmt.Errorf("error waiting for controller manager pod to expose metrics: %v; %v", err, lastMetricsFetchErr)
+                       err = fmt.Errorf("error waiting for controller manager pod to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr)
                        return
                }
        })

liggitt on Mar 12, 2020

seen again in https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/89010/pull-kubernetes-e2e-gce/1237448373842219008

liggitt on Mar 10, 2020

keeping open until https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&text=Error getting c-m metrics&job=pull-kubernetes-e2e-gce%24&test=sig-storage shows this is resolved

liggitt on Mar 10, 2020

/priority critical-urgent

bumping priority since we’re seeing significant numbers of failures that just started in the last two days

liggitt on Feb 14, 2020