alertmanager: AlertManager not sending summary or description

For completeness’ sake, this was my scenario and solution:

ALERT InstanceDown
IF up == 0
FOR 30s
ANNOTATIONS {
    summary = "Server {{ $labels.Server }} is down.",
    description = "{{ $labels.Server }} ({{ $labels.job }}) is down for more than 30 seconds."
}

receivers:
- name: general_receiver
  slack_configs:
  - api_url: https://hooks.slack.com/services/token
    channel: "#telemetry"
    title: "My summary"
    text: "My description"

{{ define "__slack_text" }}                                                                                                                                                
{{ range .Alerts }}{{ .Annotations.description}}{{ end }}                                                                                
{{ end }}

{{ define "__slack_title" }}                                                                                                                                                
{{ range .Alerts }}{{ .Annotations.summary}}{{ end }}                                                                                
{{ end }}

{{ define "slack.default.text" }}{{ template "__slack_text" . }}{{ end }}
{{ define "slack.default.title" }}{{ template "__slack_title" . }}{{ end }}

@Vannevelj, @asbjornenge, sorry guys, I was thinking you would go for comments, anyway:

I did like this, nothing fancy, for now it works for me:

title: '{{ .CommonAnnotations.summary }}'
text: '{{ .CommonAnnotations.description }}'

that picks my summary and description in prometheus’ alert.rules

@cvielma Unfortunately there’s no full documentation on Alertmanager notification templating yet. There’s a blog post to help you get started with the basics: https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/

For examples of what you can do, have a look at the default templates that you can override in your AM config file: https://github.com/prometheus/alertmanager/blob/master/template/default.tmpl

In case somebody else ends up here; TL;DR; make all your annotations EXACTLY equal. Avoid templating with variability on annotations (like summary or description).

As pointed out above, source code checks CommonAnnotations. Where it says “Common” it means common to all received alerts. If you are grouping several alerts with different labels within, then it will consider just the common annotations to all of them.

In my case, I had these annotations:

        annotations:
          description: DNS lookups are slow
          summary: DNS lookup's 99 percentile is {{ $value }} seconds.

Unfortunately, different grouped alerts had a different $value, which means that annotations were not exactly common to all alerts and because of that, they were removed from the CommonAnnotations array (and therefore, the array was empty).

Again, have a look at what this method does: https://github.com/prometheus/alertmanager/blob/v0.26.0/template/template.go#L345 Especially, these: https://github.com/prometheus/alertmanager/blob/v0.26.0/template/template.go#L390-L394

You can skip the template creation in @Vannevelj’s suggestion with something like:

text: "{{ range .Alerts }}**[{{ .Status }}]** {{ .Annotations.summary }}\n{{ .Annotations.description }}\n\n{{ end }}"

Which will get you something like:

[FIRING:2] filesystem_threshold_exceeded (ext4 node) [firing] server1.foo.bar:9100’s filesystem usage is dangerously high /data only has 0% free.

[resolved] server2.foo.bar:9100’s filesystem usage is dangerously high /data only has 3.247% free.

[firing] server3.foo.bar:9100’s filesystem usage is dangerously high /data only has 6.901% free.

More useful info found in my Googlings:

• Default template vars
• Alert struct definition

If you are using confd, then you need to escape the curly braces: Ex: name: 'slack_chatbots'slack_configs: - send_resolved: true api_url: '<url of slack web hook>' channel: '#test' text: 'https://internal.myorg.net/wiki/alerts/{{"{{"}} .GroupLabels.alertname {{"}}"}}'