prometheus-engine: Managed alertmanager no longer sending slack alerts

Hello,

We have noticed that since March 18th, our managed alertmanager is no longer sending alerts. We hve not made any configuration changes to the alertmanager config contained in the alertmanager secret, and we can see that the configuration is loaded correctly.

Alertmanager logs are full of errors now however:

{"caller":"main.go:192","level":"info","msg":"Starting Alertmanager","ts":"2025-03-22T08:25:10.555Z","version":"(version=0.27.0-gmp.1, branch=, revision=0ddd406d04a5076cb73567d5a11972bfefc7e833)"}
{"build_context":"(go=go1.22.7 X:boringcrypto, platform=linux/amd64, user=, date=2024-09-20T12:17:58+00:00, tags=boring)","caller":"main.go:193","level":"info","ts":"2025-03-22T08:25:10.555Z"}
{"caller":"cluster.go:683","component":"cluster","interval":"2s","level":"info","msg":"Waiting for gossip to settle...","ts":"2025-03-22T08:25:10.556Z"}
{"caller":"coordinator.go:113","component":"configuration","file":"/alertmanager/config_out/config.yaml","level":"info","msg":"Loading configuration file","ts":"2025-03-22T08:25:10.593Z"}
{"caller":"coordinator.go:126","component":"configuration","file":"/alertmanager/config_out/config.yaml","level":"info","msg":"Completed loading of configuration file","ts":"2025-03-22T08:25:10.594Z"}
{"address":"[::]:9093","caller":"tls_config.go:313","level":"info","msg":"Listening on","ts":"2025-03-22T08:25:10.597Z"}
{"address":"[::]:9093","caller":"tls_config.go:316","http2":false,"level":"info","msg":"TLS is disabled.","ts":"2025-03-22T08:25:10.597Z"}
{"caller":"coordinator.go:113","component":"configuration","file":"/alertmanager/config_out/config.yaml","level":"info","msg":"Loading configuration file","ts":"2025-03-22T08:25:11.659Z"}
{"caller":"coordinator.go:126","component":"configuration","file":"/alertmanager/config_out/config.yaml","level":"info","msg":"Completed loading of configuration file","ts":"2025-03-22T08:25:11.661Z"}
{"before":0,"caller":"cluster.go:708","component":"cluster","elapsed":"2.000733581s","level":"info","msg":"gossip not settled","now":1,"polls":0,"ts":"2025-03-22T08:25:12.557Z"}
{"caller":"cluster.go:700","component":"cluster","elapsed":"10.004375193s","level":"info","msg":"gossip settled; proceeding","ts":"2025-03-22T08:25:20.561Z"}
{"aggrGroup":"{}/{airflow=\"true\"}:{alertname=\"Airflow DAG failures\"}","attempts":1,"caller":"notify.go:848","component":"dispatcher","err":"Post \"<redacted>\": unsupported protocol scheme \"\"","integration":"slack[0]","level":"warn","msg":"Notify attempt failed, will retry later","receiver":"airflow-slack-notifications","ts":"2025-03-22T14:40:35.220Z"}
{"caller":"dispatch.go:353","component":"dispatcher","err":"airflow-slack-notifications/slack[0]: notify retry canceled after 15 attempts: Post \"<redacted>\": unsupported protocol scheme \"\"","level":"error","msg":"Notify for alerts failed","num_alerts":1,"ts":"2025-03-22T14:45:35.216Z"}
{"aggrGroup":"{}/{airflow=\"true\"}:{alertname=\"Airflow DAG failures\"}","attempts":1,"caller":"notify.go:848","component":"dispatcher","err":"Post \"<redacted>\": unsupported protocol scheme \"\"","integration":"slack[0]","level":"warn","msg":"Notify attempt failed, will retry later","receiver":"airflow-slack-notifications","ts":"2025-03-22T14:45:35.217Z"}
{"caller":"dispatch.go:353","component":"dispatcher","err":"airflow-slack-notifications/slack[0]: notify retry canceled after 17 attempts: Post \"<redacted>\": unsupported protocol scheme \"\"","level":"error","msg":"Notify for alerts failed","num_alerts":1,"ts":"2025-03-22T14:50:35.217Z"}

I imagine this is somehow related to the 0.27.0 update but I am unable to tell where the problem is in our config since it gets loaded correctly. I also cannot find any breaking change announcements in the docs about this changing in some way. Can you please provide more guidance?

Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 18

Commits related to this issue

Most upvoted comments

I can confirm that adding the google_cloud.external_url section in our alertmanager secret has resolved the situation for now and we are once again getting alerts.

Anyway, we blocked faulty version rollouts and we are slowly rolling out fixed 0.14 and 0.15 versions.

And for more investigation, I went on a git diff hunt git diff v0.13.1..v0.14.0.

I suspect #1074 to be the problem. I’m not that familar with the code base, but my guess:

TL;DR: Previously alertmanager config was taken from the secret in gmp-public and used directly. #1074 uses the prometheus engine config unmarshaller to parse the bytes from the provided config, and that has security measures to ensure that secrets aren’t leaked. Unfortunately that means any “secret” field that was in the secret is being obsfucated.

Note: I’m not a go expert, but just stepping through the logic to see what changed and what might be the cause, so take the above with a large grain of salt.

Either way, I don’t know if there are any workarounds. I suspect we can’t use the global.slack_api_url_file (type <filepath>) because the operatorconfig doesn’t give us the ability to write any other files sadly.

And to be thorough, I mounted a debug container on the running alertmanager-0 pod with the same volume mounts, I can confirm that /alertmanager/config/config.yaml has the same contents as the secret (and if the secret changes, the values on disk also change).

And another find, if I add a value into global.smtp_auth_password, which is also a default <secret> field per the [AM docs] https://prometheus.io/docs/alerting/latest/configuration/#file-layout-and-global-settings), then that field also shows up as <secret>. e.g. Assumption is that any field in the AM config that is <secret> per the docs is not being rendered correctly.

We’re also experiencing something similar whilst using GKE. Below are some details from a ticket we raised with google support:

Setup

Generate a “valid” alertmanager.yaml:

cat <<EOF > alertmanager.yaml
route:
  receiver: 'slack'
receivers:
- name: 'slack'
  slack_configs:
  - channel: '#some_channel'
    api_url: https://slack.com/api/chat.postMessage
    http_config:
      authorization:
        type: 'Bearer'
        credentials: 'redacted'
EOF

(Note: ‘redacted’ was the true slack token if)

Apply to the cluster (per the docs):

kubectl create secret generic alertmanager -n gmp-public --from-file=alertmanager.yaml

Get the resultant secret from the gmp-system namespace to show configuration error

Check: This shell command will get the resultant alertmanager config: kubectl get -n gmp-system secrets/alertmanager --template='{{index .data "config.yaml"}}' | base64 -d

Result on v1.30.x GKE cluster which runs v0.13

route:
  receiver: 'slack'
receivers:
- name: 'slack'
  slack_configs:
  - channel: '#some_channel'
    api_url: https://slack.com/api/chat.postMessage
    http_config:
      authorization:
        type: 'Bearer'
        credentials: 'redacted'

This worked for us previously

Result on v1.31.x GKE cluster which runs v0.14

global:
   resolve_timeout: 5m
   http_config:
       follow_redirects: true
       enable_http2: true
   smtp_hello: localhost
   smtp_require_tls: true
   pagerduty_url: https://events.pagerduty.com/v2/enqueue
   opsgenie_api_url: https://api.opsgenie.com/
   wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
   victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
   telegram_api_url: https://api.telegram.org
   webex_api_url: https://webexapis.com/v1/messages
route:
   receiver: slack
   continue: false
receivers:
   - name: slack
     slack_configs:
       - send_resolved: false
         http_config:
           authorization:
               type: Bearer
               credentials: <secret>
           follow_redirects: true
           enable_http2: true
         api_url: <secret>
         channel: '#ops-alerts-dev'
         username: '{{ template "slack.default.username" . }}'
         color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
         title: '{{ template "slack.default.title" . }}'
         title_link: '{{ template "slack.default.titlelink" . }}'
         pretext: '{{ template "slack.default.pretext" . }}'
         text: '{{ template "slack.default.text" . }}'
         short_fields: false
         footer: '{{ template "slack.default.footer" . }}'
         fallback: '{{ template "slack.default.fallback" . }}'
         callback_id: '{{ template "slack.default.callbackid" . }}'
         icon_emoji: '{{ template "slack.default.iconemoji" . }}'
         icon_url: '{{ template "slack.default.iconurl" . }}'
         link_names: false
templates: []

Notes:

  • Lots more detail values - generally fine/good
  • receivers[0].slack_configs[0].http_config.authorization.credentials is now the fixed string <secret> 👈 I assume this is where it is breaking.

More note: If the global.slack_api_url is provided in the alertmanager config secret, then that also gets replaced with <secret>