kubernetes: [Failing Test] [sig-storage] ci-kubernetes-e2e-gci-gce-serial

Which jobs are failing:

ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial)

Which test(s) are failing:

Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create unbound pvc count metrics for pvc controller after creating pvc only
Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create unbound pv count metrics for pvc controller after creating pv only
Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics should create prometheus metrics for volume provisioning and attach/detach
Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create total pv count metrics for with plugin and volume mode labels after creating pv
Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics should create prometheus metrics for volume provisioning errors [Slow]
Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create none metrics for pvc controller before creating any PV or PVC
Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create bound pv/pvc count metrics for pvc controller after creating both pv and pvc
e2e.go: Test

Since when has it been failing:

2021-05-11 19:20:26 +0000 UTC

Testgrid link:

https://testgrid.k8s.io/sig-release-master-informing#gce-cos-master-serial

Reason for failure:

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:526
Jun 14 15:15:06.169: Error getting c-m metricValues: error waiting for controller manager pod to expose metrics: timed out waiting for the condition; the server could not find the requested resource (get pods metrics-proxy:10257)
Unexpected error:
    <*errors.errorString | 0xc00322d500>: {
        s: "error waiting for controller manager pod to expose metrics: timed out waiting for the condition; the server could not find the requested resource (get pods metrics-proxy:10257)",
    }
    error waiting for controller manager pod to expose metrics: timed out waiting for the condition; the server could not find the requested resource (get pods metrics-proxy:10257)
occurred

Anything else we need to know:

/sig storage

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 23 (20 by maintainers)

Most upvoted comments

I think that this will be fixed by https://github.com/kubernetes/kubernetes/pull/102050 cc @pohly

aojea on Jun 22, 2021

All of the storage metric related failures are green now, the remaining storage tests that are failing are tracked in https://github.com/kubernetes/kubernetes/issues/103651

mauriciopoppe on Jul 23, 2021

/reopen

I’ll check the results in CI and also cherrypick to 1.22, then we should be able to close this issue.

mauriciopoppe on Jul 22, 2021

The sync.Once function are used for waiting for the corresponding pods once, https://github.com/kubernetes/kubernetes/pull/88982

I think it is a mistake, and the logic to obtain metrics should be out of the sync.Once function

			output, lastMetricsFetchErr = g.getSecureMetricsFromPod(g.kubeControllerManager, metav1.NamespaceSystem, kubeControllerManagerPort)

https://github.com/kubernetes/kubernetes/pull/101895#commitcomment-50902953

aojea on Jul 20, 2021

@ramrodo @kubernetes/sig-storage has a meeting in ~30 minutes and I ~~passively aggressively added it to their agenda~~ politely asked to review failures 😃

lambdanis on Jul 15, 2021

Now we just need to convince someone to merge that PR…

pohly on Jun 22, 2021