kyverno: [BUG] kyverno initContainer fail to start when external.metrics.k8s.io/v1beta1 return empty array for resources

Software version numbers State the version numbers of applications involved in the bug.

Kubernetes version: 1.16
Kyverno version: v1.3.0 and v1.3.1

Describe the bug A clear and concise description of what the bug is. Kyverno init container fail to start with external metrics enable

To Reproduce Steps to reproduce the behavior:

install prometheus external metrics
install kyverno v1.3.0 or v1.3.1
kyverno pod status is Init:CrashLoopBackOff
See error ``I0121 23:21:03.940127 1 main.go:140] “msg”=“Using in-cluster configuration” E0121 23:21:03.967938 1 memcache.go:206] couldn’t get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1 E0121 23:21:03.968323 1 client.go:337] dclient “msg”=“failed to get registered preferred resources” “error”=“unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1” I0121 23:21:03.968343 1 client.go:286] dclient “msg”=“schema not found” “kind”=“ValidatingWebhookConfiguration” E0121 23:21:03.968727 1 client.go:337] dclient “msg”=“failed to get registered preferred resources” “error”=“unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1” I0121 23:21:03.968755 1 client.go:286] dclient “msg”=“schema not found” “kind”=“ValidatingWebhookConfiguration” E0121 23:21:03.970418 1 client.go:337] dclient “msg”=“failed to get registered preferred resources” “error”=“unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1” I0121 23:21:03.970443 1 client.go:286] dclient “msg”=“schema not found” “kind”=“MutatingWebhookConfiguration” E0121 23:21:03.970578 1 client.go:337] dclient “msg”=“failed to get registered preferred resources” “error”=“unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1” I0121 23:21:03.970592 1 client.go:286] dclient “msg”=“schema not found” “kind”=“MutatingWebhookConfiguration” E0121 23:21:03.971480 1 client.go:337] dclient “msg”=“failed to get registered preferred resources” “error”=“unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1” I0121 23:21:03.971498 1 client.go:286] dclient “msg”=“schema not found” “kind”=“ValidatingWebhookConfiguration” E0121 23:21:03.971788 1 client.go:337] dclient “msg”=“failed to get registered preferred resources” “error”=“unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1” I0121 23:21:03.971799 1 client.go:286] dclient “msg”=“schema not found” “kind”=“ValidatingWebhookConfiguration” E0121 23:21:03.972535 1 client.go:337] dclient “msg”=“failed to get registered preferred resources” “error”=“unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1” I0121 23:21:03.972554 1 client.go:286] dclient “msg”=“schema not found” “kind”=“MutatingWebhookConfiguration” E0121 23:21:03.973374 1 client.go:337] dclient “msg”=“failed to get registered preferred resources” “error”=“unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1” I0121 23:21:03.973395 1 client.go:286] dclient “msg”=“schema not found” “kind”=“MutatingWebhookConfiguration” E0121 23:21:03.973791 1 client.go:337] dclient “msg”=“failed to get registered preferred resources” “error”=“unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1” I0121 23:21:03.973804 1 client.go:286] dclient “msg”=“schema not found” “kind”=“Namespace” E0121 23:21:03.974495 1 client.go:337] dclient “msg”=“failed to get registered preferred resources” “error”=“unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1” I0121 23:21:03.974519 1 client.go:286] dclient “msg”=“schema not found” “kind”=“ClusterPolicyReport” E0121 23:21:03.974961 1 client.go:337] dclient “msg”=“failed to get registered preferred resources” “error”=“unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1” I0121 23:21:03.974996 1 client.go:286] dclient “msg”=“schema not found” “kind”=“ReportChangeRequest” E0121 23:21:03.975570 1 client.go:337] dclient “msg”=“failed to get registered preferred resources” “error”=“unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1” I0121 23:21:03.975586 1 client.go:286] dclient “msg”=“schema not found” “kind”=“ClusterReportChangeRequest” panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x115b350]

goroutine 25 [running]: main.removeReportChangeRequest(0xc00007f220, 0x13bfe86, 0x13, 0xc000082901, 0xc000397ef0) /home/runner/work/kyverno/kyverno/cmd/initContainer/main.go:312 +0x110 main.executeRequest(0xc00007f220, 0x13bfe86, 0x13, 0x13b48e8, 0x0, 0x0, 0x0) /home/runner/work/kyverno/kyverno/cmd/initContainer/main.go:128 +0x179 main.process.func1(0xc0000829c0, 0xc000082960, 0xc00007f220, 0xc000082900, 0x158b720, 0xc000325ec0, 0xc0000825a0) /home/runner/work/kyverno/kyverno/cmd/initContainer/main.go:188 +0xfb created by main.process /home/runner/work/kyverno/kyverno/cmd/initContainer/main.go:184 +0xd7 5. kubectl get --raw “/apis/external.metrics.k8s.io/v1beta1” | jq { “kind”: “APIResourceList”, “apiVersion”: “v1”, “groupVersion”: “external.metrics.k8s.io/v1beta1”, “resources”: [] }` 6. delete external metrics api, redeploy kyverno, it starts successful.

Expected behavior A clear and concise description of what you expected to happen. external metrics return empty resource which is correct when there is no metrics. Kyverno shouldn’t fail with empty resource.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 2
Comments: 35 (14 by maintainers)

Commits related to this issue

fixes https://github.com/kyverno/kyverno/issues/1490 Signed-off-by: Shuting Zhao <shutting06@gmail.com> — committed to realshuting/kyverno by realshuting 3 years ago
Refactor resourceCache; Reduce throttling requests (background controller) (#1500) * skip sending API request for filtered resource * fix PR comment Signed-off-by: Shuting Zhao <shutting06@gmai... — committed to kyverno/kyverno by realshuting 3 years ago
Refactor resourceCache; Reduce throttling requests (background controller) (#1500) * skip sending API request for filtered resource * fix PR comment Signed-off-by: Shuting Zhao <shutting06@gmail.co... — committed to kyverno/kyverno by realshuting 3 years ago
Refactor resourceCache; Reduce throttling requests (background controller) (#1500) * skip sending API request for filtered resource * fix PR comment Signed-off-by: Shuting Zhao <shutting06@gmail.co... — committed to kyverno/kyverno by realshuting 3 years ago
api server lookups (#1514) * initial commit for api server lookups Signed-off-by: Jim Bugwadia <jim@nirmata.com> * initial commit for API server lookups Signed-off-by: Jim Bugwadia <jim@nirm... — committed to kyverno/kyverno by JimBugwadia 3 years ago
Switch to use annotations to store resource info in cluster/reportChangeRequest (#1625) * skip sending API request for filtered resource * fix PR comment Signed-off-by: Shuting Zhao <shutting06... — committed to kyverno/kyverno by realshuting 3 years ago
Switch to use annotations to store resource info in cluster/reportChangeRequest (#1625) * skip sending API request for filtered resource * fix PR comment Signed-off-by: Shuting Zhao <shutting06... — committed to vyankyGH/kyverno by realshuting 3 years ago

Most upvoted comments

There should be no reason why Kyverno depends on the custom metrics endpoint actually having a metric behind it. Right?

Ideally we want to ignore such errors. Adding to milestone 1.6.0.

I would argue that there is no error at all … it’s totally reasonable to have an APIService that is not currently exposing any metrics, but is responsive.

diranged on Oct 28, 2021

Having the same issue, any ETA on this?

Amr-Aly on Nov 21, 2021

I would argue that there is no error at all … it’s totally reasonable to have an APIService that is not currently exposing any metrics, but is responsive.

Experiencing the same here, fresh Kyverno install.

Error: "msg"="sync failed, unable to update in-cluster api versions" "error"="unable to fetch apiResourceLists: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"

Relevant info:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": []
}

elchesco on Oct 29, 2021

Can confirm that release v2.1.4-rc3 did fix these issues for me and I was able to install policies to the cluster again. Thank you all!

roeelandesman on Dec 3, 2021

Thanks @roeelandesman! I tried again in 1.5.1 and was able to reproduce what you are seeing.

This was fixed in 1.6.x (main) via: https://github.com/kyverno/kyverno/pull/2634.

We will merge this fix for 1.5.2.

JimBugwadia on Dec 2, 2021

There should be no reason why Kyverno depends on the custom metrics endpoint actually having a metric behind it. Right?

Ideally we want to ignore such errors. Adding to milestone 1.6.0.

realshuting on Oct 28, 2021

@realshuting we have install an external metrics service, when there is no HPA defined, the resource return from external metrics API is empty array which is correct behavior. we can’t delete the external metrics API, since it will be used for our HPA.

nathanwang-comp on Jan 22, 2021

the workaround doesn’t fit my problem, we can’t delete the api, since we have the external metrics adapter.

nathanwang-comp on Jan 22, 2021