prometheus-operator: Prometheus Helm Chart no longer has permission to talk to pods in Kube-System

What did you do? Installed the Chart using Helm. Default Yaml, no amendments.

What did you expect to see? Prometheus pulling metrics from all namespaces

What did you see instead? Under which circumstances? Prometheus has access to all namespaces except kube-system.

Environment Centos 7

Kubernetes version information: Client Version: version.Info{Major:“1”, Minor:“10”, GitVersion:“v1.10.2”, GitCommit:“81753b10df112992bf51bbc2c2f85208aad78335”, GitTreeState:“clean”, BuildDate:“2018-04-27T09:22:21Z”, GoVersion:“go1.9.3”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“10”, GitVersion:“v1.10.2”, GitCommit:“81753b10df112992bf51bbc2c2f85208aad78335”, GitTreeState:“clean”, BuildDate:“2018-04-27T09:10:24Z”, GoVersion:“go1.9.3”, Compiler:“gc”, Platform:“linux/amd64”}
Prometheus Operator Logs:

endpoint="https-metrics" instance="10.26.3.63:10250" namespace="kube-system" service="kubelet"
server returned HTTP status 403 Forbidden

The Helm chart was working till K8s was updated to 1.10.2. I removed the chart, re-added the image, and installed the chart. All looked good, till I checked Prometheus and showed the above error.

If there is anything else you need, please let me know.

Regards,

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 17 (14 by maintainers)

Most upvoted comments

Made the below additional config to resolve the issue.

$ cat /etc/systemd/system/kubelet.service.d/01-kubeadm.conf

Ansible managed

[Service] Environment=“KUBELET_EXTRA_ARGS=–authentication-token-webhook”

shankerbalan on Jun 2, 2018