prometheus-engine: Managed alertmanager no longer sending slack alerts
Hello,
We have noticed that since March 18th, our managed alertmanager is no longer sending alerts. We hve not made any configuration changes to the alertmanager config contained in the alertmanager secret, and we can see that the configuration is loaded correctly.
Alertmanager logs are full of errors now however:
{"caller":"main.go:192","level":"info","msg":"Starting Alertmanager","ts":"2025-03-22T08:25:10.555Z","version":"(version=0.27.0-gmp.1, branch=, revision=0ddd406d04a5076cb73567d5a11972bfefc7e833)"}
{"build_context":"(go=go1.22.7 X:boringcrypto, platform=linux/amd64, user=, date=2024-09-20T12:17:58+00:00, tags=boring)","caller":"main.go:193","level":"info","ts":"2025-03-22T08:25:10.555Z"}
{"caller":"cluster.go:683","component":"cluster","interval":"2s","level":"info","msg":"Waiting for gossip to settle...","ts":"2025-03-22T08:25:10.556Z"}
{"caller":"coordinator.go:113","component":"configuration","file":"/alertmanager/config_out/config.yaml","level":"info","msg":"Loading configuration file","ts":"2025-03-22T08:25:10.593Z"}
{"caller":"coordinator.go:126","component":"configuration","file":"/alertmanager/config_out/config.yaml","level":"info","msg":"Completed loading of configuration file","ts":"2025-03-22T08:25:10.594Z"}
{"address":"[::]:9093","caller":"tls_config.go:313","level":"info","msg":"Listening on","ts":"2025-03-22T08:25:10.597Z"}
{"address":"[::]:9093","caller":"tls_config.go:316","http2":false,"level":"info","msg":"TLS is disabled.","ts":"2025-03-22T08:25:10.597Z"}
{"caller":"coordinator.go:113","component":"configuration","file":"/alertmanager/config_out/config.yaml","level":"info","msg":"Loading configuration file","ts":"2025-03-22T08:25:11.659Z"}
{"caller":"coordinator.go:126","component":"configuration","file":"/alertmanager/config_out/config.yaml","level":"info","msg":"Completed loading of configuration file","ts":"2025-03-22T08:25:11.661Z"}
{"before":0,"caller":"cluster.go:708","component":"cluster","elapsed":"2.000733581s","level":"info","msg":"gossip not settled","now":1,"polls":0,"ts":"2025-03-22T08:25:12.557Z"}
{"caller":"cluster.go:700","component":"cluster","elapsed":"10.004375193s","level":"info","msg":"gossip settled; proceeding","ts":"2025-03-22T08:25:20.561Z"}
{"aggrGroup":"{}/{airflow=\"true\"}:{alertname=\"Airflow DAG failures\"}","attempts":1,"caller":"notify.go:848","component":"dispatcher","err":"Post \"<redacted>\": unsupported protocol scheme \"\"","integration":"slack[0]","level":"warn","msg":"Notify attempt failed, will retry later","receiver":"airflow-slack-notifications","ts":"2025-03-22T14:40:35.220Z"}
{"caller":"dispatch.go:353","component":"dispatcher","err":"airflow-slack-notifications/slack[0]: notify retry canceled after 15 attempts: Post \"<redacted>\": unsupported protocol scheme \"\"","level":"error","msg":"Notify for alerts failed","num_alerts":1,"ts":"2025-03-22T14:45:35.216Z"}
{"aggrGroup":"{}/{airflow=\"true\"}:{alertname=\"Airflow DAG failures\"}","attempts":1,"caller":"notify.go:848","component":"dispatcher","err":"Post \"<redacted>\": unsupported protocol scheme \"\"","integration":"slack[0]","level":"warn","msg":"Notify attempt failed, will retry later","receiver":"airflow-slack-notifications","ts":"2025-03-22T14:45:35.217Z"}
{"caller":"dispatch.go:353","component":"dispatcher","err":"airflow-slack-notifications/slack[0]: notify retry canceled after 17 attempts: Post \"<redacted>\": unsupported protocol scheme \"\"","level":"error","msg":"Notify for alerts failed","num_alerts":1,"ts":"2025-03-22T14:50:35.217Z"}
I imagine this is somehow related to the 0.27.0 update but I am unable to tell where the problem is in our config since it gets loaded correctly. I also cannot find any breaking change announcements in the docs about this changing in some way. Can you please provide more guidance?
Thanks!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 18
Commits related to this issue
- tests: add regression test for #1550 (AM + secrets bug) Signed-off-by: bwplotka <bwplotka@google.com> — committed to GoogleCloudPlatform/prometheus-engine by bwplotka 2 years ago
- test: add regression test for #1550 (AM + secrets bug) Signed-off-by: bwplotka <bwplotka@google.com> — committed to GoogleCloudPlatform/prometheus-engine by bwplotka 2 years ago
- test: add regression unit test against #1550 https://github.com/GoogleCloudPlatform/prometheus-engine/issues/1550 Signed-off-by: bwplotka <bwplotka@google.com> — committed to GoogleCloudPlatform/prometheus-engine by bwplotka 2 years ago
- test: add regression unit test against #1550 https://github.com/GoogleCloudPlatform/prometheus-engine/issues/1550 Signed-off-by: bwplotka <bwplotka@google.com> — committed to GoogleCloudPlatform/prometheus-engine by bwplotka 2 years ago
- test: add regression unit test against #1550 https://github.com/GoogleCloudPlatform/prometheus-engine/issues/1550 Signed-off-by: bwplotka <bwplotka@google.com> — committed to GoogleCloudPlatform/prometheus-engine by bwplotka 2 years ago
- test: add regression unit test against #1550 https://github.com/GoogleCloudPlatform/prometheus-engine/issues/1550 Signed-off-by: bwplotka <bwplotka@google.com> — committed to GoogleCloudPlatform/prometheus-engine by bwplotka 2 years ago
- test: add regression unit test against #1550 https://github.com/GoogleCloudPlatform/prometheus-engine/issues/1550 Signed-off-by: bwplotka <bwplotka@google.com> — committed to GoogleCloudPlatform/prometheus-engine by bwplotka 2 years ago
- test: add regression unit test against #1550 https://github.com/GoogleCloudPlatform/prometheus-engine/issues/1550 Signed-off-by: bwplotka <bwplotka@google.com> — committed to GoogleCloudPlatform/prometheus-engine by bwplotka 2 years ago
- test: add regression unit test against #1550 https://github.com/GoogleCloudPlatform/prometheus-engine/issues/1550 Signed-off-by: bwplotka <bwplotka@google.com> — committed to GoogleCloudPlatform/prometheus-engine by bwplotka 2 years ago
- test: add regression unit test against #1550 https://github.com/GoogleCloudPlatform/prometheus-engine/issues/1550 Signed-off-by: bwplotka <bwplotka@google.com> — committed to GoogleCloudPlatform/prometheus-engine by bwplotka 2 years ago
- test: add regression unit test against #1550 https://github.com/GoogleCloudPlatform/prometheus-engine/issues/1550 Signed-off-by: bwplotka <bwplotka@google.com> — committed to GoogleCloudPlatform/prometheus-engine by bwplotka 2 years ago
- test: add regression unit test against #1550 https://github.com/GoogleCloudPlatform/prometheus-engine/issues/1550 Signed-off-by: bwplotka <bwplotka@google.com> — committed to GoogleCloudPlatform/prometheus-engine by bwplotka 2 years ago
I can confirm that adding the
google_cloud.external_urlsection in our alertmanager secret has resolved the situation for now and we are once again getting alerts.Anyway, we blocked faulty version rollouts and we are slowly rolling out fixed 0.14 and 0.15 versions.
And for more investigation, I went on a git diff hunt
git diff v0.13.1..v0.14.0.I suspect #1074 to be the problem. I’m not that familar with the code base, but my guess:
TL;DR: Previously alertmanager config was taken from the secret in gmp-public and used directly. #1074 uses the prometheus engine config unmarshaller to parse the bytes from the provided config, and that has security measures to ensure that secrets aren’t leaked. Unfortunately that means any “secret” field that was in the secret is being obsfucated.
Note: I’m not a go expert, but just stepping through the logic to see what changed and what might be the cause, so take the above with a large grain of salt.
Either way, I don’t know if there are any workarounds. I suspect we can’t use the
global.slack_api_url_file(type <filepath>) because the operatorconfig doesn’t give us the ability to write any other files sadly.And to be thorough, I mounted a debug container on the running
alertmanager-0pod with the same volume mounts, I can confirm that/alertmanager/config/config.yamlhas the same contents as the secret (and if the secret changes, the values on disk also change).And another find, if I add a value into
global.smtp_auth_password, which is also a default<secret>field per the [AM docs] https://prometheus.io/docs/alerting/latest/configuration/#file-layout-and-global-settings), then that field also shows up as<secret>. e.g. Assumption is that any field in the AM config that is<secret>per the docs is not being rendered correctly.We’re also experiencing something similar whilst using GKE. Below are some details from a ticket we raised with google support:
Setup
Generate a “valid” alertmanager.yaml:
(Note: ‘redacted’ was the true slack token if)
Apply to the cluster (per the docs):
kubectl create secret generic alertmanager -n gmp-public --from-file=alertmanager.yamlGet the resultant secret from the gmp-system namespace to show configuration error
Check: This shell command will get the resultant alertmanager config:
kubectl get -n gmp-system secrets/alertmanager --template='{{index .data "config.yaml"}}' | base64 -dResult on v1.30.x GKE cluster which runs v0.13
This worked for us previously
Result on v1.31.x GKE cluster which runs v0.14
Notes:
receivers[0].slack_configs[0].http_config.authorization.credentialsis now the fixed string<secret>👈 I assume this is where it is breaking.More note: If the
global.slack_api_urlis provided in the alertmanager config secret, then that also gets replaced with<secret>