metrics-server: 403 Forbidden", response: "Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=stats)

Trying to run metrics-server on a 1.10.3 cluster. With the stock yaml, I’m getting this in the logs.

E0805 16:38:05.120665 1 summary.go:97] error while getting metrics summary from Kubelet xxxxxxxxxxxxxxxx(xxxxxxxxxxx:10255): Get http://xxxxxxxxxx:10255/stats/summary/: dial tcp xxxxxxxxxx:10255: getsockopt: connection refused I found references to this error elsewhere and modified the deployment args

From: - --source=kubernetes.summary_api:‘’ To : --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

That results in errors like this

E0806 19:56:05.269962 1 summary.go:97] error while getting metrics summary from Kubelet xxxxxxxxxx(xxxxxxx:10250): request failed - "403 Forbidden", response: "Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=stats)"

How do I get the stats-server working?

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 20 (3 by maintainers)

Most upvoted comments

In metric-server 0.3.1 perhaps solve the problem

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
  - deployments
  verbs:
  - get
  - list
  - watch

TracyBin on Oct 12, 2018

Can not make it work on 1.10.4 neither with the same error 😦

lingxiankong on Aug 29, 2018

@aisensiy as I said, it was a network issue that the routes weren’t properly set up. The problem and solution were custom to my environment.

benmathews on Nov 8, 2018

also, are you starting your kubelet with --authentication-token-webhook?

liggitt on Sep 12, 2018

There are three paths for kubelet serving certs:

Created out of band, explicitly provided to the kubelet. What these certs is up to the deployer.
Self-signed by the kubelet on start. Since these are self-signed, you’d have to disable tls verification no matter what SANs the cert contained.
requested by the kubelet from the CSR API (alpha in 1.11, beta in 1.12, must be paired with an approval process for the CSR requests). The cert is requested with all the addresses the kubelet would report in its Node status.addresses field. As long as one of the address in the Node status is used to contact the node, the certificate should have a matching SAN.

liggitt on Aug 15, 2018