prometheus-adapter: Unauthorized error seen in logs

I’ve deployed prometheus and prometheus-adapter with their respective helm charts in the same namespace. The adapter is deployed with:

helm install --namespace monitoring --name prometheus-adapter stable/prometheus-adapter --set prometheus.url=http://prometheus-server --set prometheus.port=80 --set logLevel=6

Everything seems to be working, but every once in a while I see an unauthorized error with no reason specified in my adapter logs:

I0315 12:36:19.647959       1 adapter.go:91] successfully using in-cluster auth
I0315 12:36:19.666785       1 round_trippers.go:405] GET https://100.64.0.1:443/api?timeout=32s 200 OK in 17 milliseconds
I0315 12:36:19.668727       1 round_trippers.go:405] GET https://100.64.0.1:443/apis?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.671025       1 round_trippers.go:405] GET https://100.64.0.1:443/api/v1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.672989       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/apiregistration.k8s.io/v1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.674484       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/apiregistration.k8s.io/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.676083       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/extensions/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.677542       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/apps/v1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.679008       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/apps/v1beta2?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.680424       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/apps/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.681838       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/events.k8s.io/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.683255       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/authentication.k8s.io/v1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.684540       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/authentication.k8s.io/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.685879       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/authorization.k8s.io/v1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.687280       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.688653       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/autoscaling/v1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.689981       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/autoscaling/v2beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.691299       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/batch/v1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.692608       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/batch/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.693993       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/certificates.k8s.io/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.698401       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/networking.k8s.io/v1?timeout=32s 200 OK in 4 milliseconds
I0315 12:36:19.699798       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/policy/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.701310       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/rbac.authorization.k8s.io/v1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.702699       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/rbac.authorization.k8s.io/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.704123       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/storage.k8s.io/v1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.705464       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/storage.k8s.io/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.706983       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/admissionregistration.k8s.io/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.708383       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/apiextensions.k8s.io/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.709768       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/scheduling.k8s.io/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.711195       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/certmanager.k8s.io/v1alpha1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.712651       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/helm.integrations.flux.weave.works/v1alpha2?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.714025       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/openfaas.com/v1alpha2?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.718064       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/admission.certmanager.k8s.io/v1beta1?timeout=32s 200 OK in 3 milliseconds
I0315 12:36:19.719501       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/flux.weave.works/v1beta1?timeout=32s 200 OK in 1 milliseconds
I0315 12:36:19.726162       1 round_trippers.go:405] GET https://100.64.0.1:443/apis/custom.metrics.k8s.io/v1beta1?timeout=32s 200 OK in 6 milliseconds
I0315 12:36:20.193555       1 api.go:74] GET http://prometheus-server:80/api/v1/series?match%5B%5D=%7Bnamespace%21%3D%22%22%2C__name__%21~%22%5Econtainer_.%2A%22%7D&start=1552652179.735 200 OK
I0315 12:36:20.421931       1 api.go:74] GET http://prometheus-server:80/api/v1/series?match%5B%5D=%7B__name__%3D~%22%5Econtainer_.%2A%22%2Ccontainer_name%21%3D%22POD%22%2Cnamespace%21%3D%22%22%2Cpod_name%21%3D%22%22%7D&start=1552652179.735 200 OK
I0315 12:36:20.565144       1 serving.go:273] Generated self-signed cert (/tmp/cert/apiserver.crt, /tmp/cert/apiserver.key)
I0315 12:36:22.175642       1 round_trippers.go:405] GET https://100.64.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 2 milliseconds
I0315 12:36:22.180156       1 round_trippers.go:405] GET https://100.64.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 2 milliseconds
I0315 12:36:22.183447       1 round_trippers.go:405] GET https://100.64.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 2 milliseconds
I0315 12:36:22.189090       1 round_trippers.go:405] GET https://100.64.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 4 milliseconds
I0315 12:36:22.191788       1 healthz.go:83] Installing healthz checkers:"ping"
I0315 12:36:22.192063       1 serve.go:96] Serving securely on [::]:6443
I0315 12:36:55.371870       1 round_trippers.go:405] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 7 milliseconds
I0315 12:36:55.374352       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0315 12:36:55.374434       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0315 12:36:55.375050       1 wrap.go:42] GET /healthz: (10.547935ms) 200 [[kube-probe/1.11] 172.20.84.63:34116]
I0315 12:36:56.795602       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0315 12:36:56.795623       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0315 12:36:56.795651       1 wrap.go:42] GET /healthz: (132.713µs) 200 [[kube-probe/1.11] 172.20.84.63:34138]
I0315 12:37:05.278346       1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
I0315 12:37:05.281046       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (2.891526ms) 200 [[kube-apiserver/v1.11.7 (linux/amd64) kubernetes/65ecaf0] 100.114.84.128:62765]
I0315 12:37:05.364383       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0315 12:37:05.364406       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0315 12:37:05.364459       1 wrap.go:42] GET /healthz: (126.277µs) 200 [[kube-probe/1.11] 172.20.84.63:34236]
I0315 12:37:06.800635       1 round_trippers.go:405] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 4 milliseconds
I0315 12:37:06.800956       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I0315 12:37:06.800972       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I0315 12:37:06.801006       1 wrap.go:42] GET /healthz: (5.005914ms) 200 [[kube-probe/1.11] 172.20.84.63:34248]
I0315 12:37:10.324994       1 round_trippers.go:405] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I0315 12:37:10.325109       1 authorization.go:73] Forbidden: "/", Reason: ""
I0315 12:37:10.325762       1 wrap.go:42] GET /: (2.364935ms) 403 [[Go-http-client/2.0] 100.114.84.128:11698]
I0315 12:37:10.790885       1 round_trippers.go:405] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I0315 12:37:10.791475       1 handler.go:143] prometheus-metrics-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 16 (3 by maintainers)

Most upvoted comments

What part of the adapter runs as ‘system:anonymous’? It makes me nervous to give blanket access to anonymous unauthenticated users.

+17

agnewp on Apr 18, 2019

I’m seeing these as well and would like to understand why before changing anything.

POST https://172.20.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews

I’m seeing some calls logged as the above succeed and some calls return 403 Forbidden:

I1007 21:40:05.707821       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I1007 21:40:05.709716       1 round_trippers.go:386] curl -k -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer ..." 'https://172.20.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I1007 21:40:05.711812       1 round_trippers.go:405] POST https://172.20.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 2 milliseconds
I1007 21:40:05.711831       1 round_trippers.go:411] Response Headers:
I1007 21:40:05.711837       1 round_trippers.go:414]     Content-Length: 424
I1007 21:40:05.711841       1 round_trippers.go:414]     Date: Mon, 07 Oct 2019 21:40:05 GMT
I1007 21:40:05.711948       1 round_trippers.go:414]     Audit-Id: b9effe85-bd0f-481f-b8b0-5da5a4458b43
I1007 21:40:05.711957       1 round_trippers.go:414]     Content-Type: application/json
I1007 21:40:05.711997       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:public-info-viewer\" of ClusterRole \"system:public-info-viewer\" to Group \"system:unauthenticated\""}}
I1007 21:40:05.712137       1 handler.go:153] prometheus-metrics-adapter: GET "/healthz" satisfied by nonGoRestful
I1007 21:40:05.712149       1 pathrecorder.go:240] prometheus-metrics-adapter: "/healthz" satisfied by exact match
I1007 21:40:05.712187       1 wrap.go:42] GET /healthz: (4.466383ms) 200 [[kube-probe/1.14+] 10.17.68.80:58458]
...
I1007 21:40:05.785410       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:resourcequota-controller","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":false}}
I1007 21:40:05.785575       1 round_trippers.go:386] curl -k -v -XPOST  -H "Authorization: Bearer ..." -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://172.20.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I1007 21:40:05.787338       1 round_trippers.go:405] POST https://172.20.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I1007 21:40:05.787355       1 round_trippers.go:411] Response Headers:
I1007 21:40:05.787361       1 round_trippers.go:414]     Content-Length: 526
I1007 21:40:05.787366       1 round_trippers.go:414]     Date: Mon, 07 Oct 2019 21:40:05 GMT
I1007 21:40:05.787371       1 round_trippers.go:414]     Audit-Id: 4c890fa3-caf7-4656-ab85-f12700078fb5
I1007 21:40:05.787376       1 round_trippers.go:414]     Content-Type: application/json
I1007 21:40:05.787399       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:resourcequota-controller","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":true,"reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
I1007 21:40:05.787480       1 handler.go:143] prometheus-metrics-adapter: GET "/apis/metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1
I1007 21:40:05.787579       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (13.635587ms) 200 [[kube-controller-manager/v1.14.6 (linux/amd64) kubernetes/5047edc/system:serviceaccount:kube-system:resourcequota-controller] 10.17.41.255:35776]

and

I1007 21:40:06.004237       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I1007 21:40:06.004521       1 round_trippers.go:386] curl -k -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer ..." 'https://172.20.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I1007 21:40:06.005960       1 round_trippers.go:405] POST https://172.20.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 1 milliseconds
I1007 21:40:06.005987       1 round_trippers.go:411] Response Headers:
I1007 21:40:06.005993       1 round_trippers.go:414]     Audit-Id: d773b903-001f-4eaf-918d-af29328657a0
I1007 21:40:06.005998       1 round_trippers.go:414]     Content-Type: application/json
I1007 21:40:06.006003       1 round_trippers.go:414]     Content-Length: 260
I1007 21:40:06.006008       1 round_trippers.go:414]     Date: Mon, 07 Oct 2019 21:40:06 GMT
I1007 21:40:06.006031       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I1007 21:40:06.006112       1 authorization.go:73] Forbidden: "/", Reason: ""
I1007 21:40:06.006199       1 wrap.go:42] GET /: (2.474734ms) 403 [[Go-http-client/2.0] 10.17.72.69:52772]

These logs seem to be from incoming calls to the prometheus adapter, not outgoing calls from the prometheus adapter; one call that succeeds as user system:anonymous seems to be for the healthcheck to url path /healthz.

@gokhandincer @s-urbaniak so what would be making incoming calls as user system:anonymous to root url path /, is it a valid url path, and is it really safe to enable system:anonymous user access?

gitfool on Oct 7, 2019

@jaydp17 in my case it didn’t work unless I give following permission to system:anonymous

- nonResourceURLs:
    - "/apis/custom.metrics.k8s.io/v1beta1"
  verbs: [ "get" ]

we don’t need to add openapi/v2 anyway it is 404, but this is just workaround. as suggested by @Ludek2 , setting proxy-client parameters would be the ideal solution.

surajnarwade on May 22, 2020