prometheus-operator: AlertmanagerConfig not getting picked up

My AlertmanagerConfig is not getting picked up by the Alertmanager. I have the CRD monitoring.coreos.com.alertmanagerconfigs.yaml version 0.46.0 applied in the cluster and I’m using kube-prometheus-stack.

Can someone guide me here?

Environment K8s EKS

Prometheus Operator version:

Image:         quay.io/coreos/prometheus-operator:v0.40.0
Image ID:      docker-pullable://quay.io/coreos/prometheus-operator@sha256:a54e806fb27d2fb0251da4f3b2a3bb5320759af63a54a755788304775f2384a7

Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Prometheus-operator logs: No logs are reflected when the below manifest is added/deleted/edited.

Manifests:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  labels:
    alerts: "true"
    namespace: o11y # <-------------- kube-prometheus-stack is also installed in the same namespace "o11y"
spec:
  receivers:
  - name: prom-poc-alerts
    slackConfigs:
    - apiURL:
        key: prom-poc-alerts
      channel: '#prom-poc-alerts'
      sendResolved: true
      text: |-
        {{ range .Alerts }}{{ .Annotations.summary }}
        {{ .Annotations.description }}
        {{ .Annotations.message }}
        {{ end }}
  route:
    groupBy:
    - job
    groupInterval: 5m
    groupWait: 60s
    receiver: prom-poc-alerts
    repeatInterval: 10h
    routes:
    - match:
        namespace: o11y
      receiver: prom-poc-alerts

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 25 (3 by maintainers)

Most upvoted comments

Would be awesome if someone can show us a working example.

At the moment I am stuck at this configuration:

  route:
    groupBy:
    - job
    groupInterval: 5m
    groupWait: 60s
    receiver: prom-poc-alerts
    repeatInterval: 10h
    matchers:
     - name: namespace
       value: o11y
       regex: false
    receiver: prom-poc-alerts

I don’t know the update on which this changed. Before I was able to create a “match” over multiple namespaces. Now it looks like a AlertmanagerConfig is forced on one namespace. https://github.com/prometheus-operator/prometheus-operator/blob/dab2194da371aee19ce251bcad34f8debb91ea9f/pkg/alertmanager/amcfg.go#L143-L146

A couple of weeks ago I was able to do a global notification configuration like:

  route:
    groupBy:
    - job
    - alertname
    receiver: some-receiver
    routes:
    - match:
        department: it
      receiver: some-receiver

PetzJohannes on Jul 6, 2021