prometheus-operator: Using bearerTokenSecret doesn't include the Bearer Token in the Authentication Header

I’m trying to use the bearerTokenSecret definition in my ServiceMonitor to scrape my target’s /metrics endpoint with a Bearer Token (used for authentication) included in the Authorization Header. It looks like my target is getting scraped, but the token I’m including in the secret I provide is not being included in the Authorization Header. The endpoint configuration for my ServiceMonitor looks something like this:

spec:
  endpoints:
    - bearerTokenSecret:
        key: token
        name: auth-token-token-6kzmt
      interval: 60s
      port: 9080-tcp
      tlsConfig:
        insecureSkipVerify: true

When I look at my application’s access logs (The format of the access log is: 1) User-Agent, 2) Authorization Header), the requests from Prometheus have nothing in the Authorization Header (represented by the dash).

Prometheus/2.7.1 -

In comparison, when I try to curl with the bearer token using -H "Authorization: Bearer <TOKEN>, I can see the Bearer Token I include in the curl request:

curl/7.29.0 Bearer asdUadf....

For other example authentication methods like the ServiceMonitor’s basicAuth definition, I can see those credentials in the Authorization Header as well:

Prometheus/2.7.1 Basic asdUadf....

Not sure why using the bearerTokenSecret definition in the ServiceMonitor doesn’t include the specified secret’s token inside the Authorization Header, and not sure if this is a configuration issue or a bug.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 7
Comments: 38 (10 by maintainers)

Most upvoted comments

My current setup is available at https://github.com/thaum-xyz/ankhmorpork/blob/master/apps/homeassistant/08_servicemonitor.yaml#L12. This is running prometheus-operator v0.40.0 and I didn’t change anything (working OOTB).

I had one issue when encoding token value as it also encoded \n and authentication failed, but this was not related to prometheus-operator.

paulfantom on Jul 23, 2020

Exactly, what I mentioned in my previous comment was the buggy version.

Wrong

valueFrom:
  secretKeyRef:
    name: "secretX"
    key: "keyX"

Corrrect

bearerTokenSecret:
  name: "secretX"
  key: "keyX"

With the correct version I was able to make it work in 0.39.0.

jizi on Jul 31, 2020

@cryptarchnoble you are using too old version. Support for bearerTokenSecret was added in https://github.com/coreos/prometheus-operator/pull/2716 which is part of 0.34.0 release.

@kanadaj OpenShift 3.11 is using prometheus-operator in version 0.23.2 (plus some patches), which is too old to support bearerTokenSecret. Source: https://github.com/openshift/prometheus-operator/blob/release-3.11/VERSION

paulfantom on Jul 27, 2020

@cryptarchnoble Since it can be repro’d on OKD 3.11, it’s unlikely to be Rancher’s fault.

kanadaj on Jul 27, 2020