prometheus: External Labels not available in alert Annotations

What did you do?

Defined an external label in prometheus config:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
  external_labels:
    monitor: acme-logs-dev

In a rule I had:

ALERT container_eating_memory
  IF sum(container_memory_rss{container_label_com_docker_swarm_task_name=~".+"}) BY (instance, name) > 2500000000
  FOR 5m
  ANNOTATIONS {
    description="{{ $labels.container_label_com_docker_swarm_task_name }} is eating up a LOT of memory. Memory consumption of {{ $labels.container_label_com_docker_swarm_task_name }} is at {{ humanize $value}}.", 
    summary="{{$labels.monitor}} - HIGH MEMORY USAGE WARNING: TASK '{{ $labels.container_label_com_docker_swarm_task_name }}' on '{{ $labels.instance }}'"}

What did you expect to see? The templated alert printing the monitor label.

But the alert in both prometheus alert view and the alert that ended up in slack were both missing the external label

What did you see instead? Under which circumstances?

No label listed in prometheus alerts page and empty string in slack

Environment

• System information:

• Prometheus version:

  1.7.1

• Alertmanager version:

  0.8.0

• Prometheus configuration file:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
  external_labels:
    monitor: acme-logs-dev
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alertmanager.service.acme:9093
    scheme: http
    timeout: 10s
rule_files:
- /etc/prometheus/tasks.rules
- /etc/prometheus/host.rules
- /etc/prometheus/containers.rules
scrape_configs:
- job_name: cadvisor
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  dns_sd_configs:
  - names:
    - tasks.cadvisor
    refresh_interval: 30s
    type: A
    port: 8080
- job_name: node-exporter
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  dns_sd_configs:
  - names:
    - tasks.nodeexporter
    refresh_interval: 30s
    type: A
    port: 9100
- job_name: prometheus
  scrape_interval: 10s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - localhost:9090
- job_name: blackbox-http
  params:
    module:
    - http_2xx
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /probe
  scheme: http
  dns_sd_configs:
  - names:
    - tasks.kibana.service.acme
    refresh_interval: 30s
    type: A
    port: 5601
  relabel_configs:
  - source_labels: [__address__]
    separator: ;
    regex: (.*)(:80)?
    target_label: __param_target
    replacement: ${1}
    action: replace
  - source_labels: [__param_target]
    separator: ;
    regex: (.*)
    target_label: instance
    replacement: ${1}
    action: replace
  - source_labels: []
    separator: ;
    regex: .*
    target_label: __address__
    replacement: blackboxexporter:9115
    action: replace
  - source_labels: [__meta_dns_name]
    separator: ;
    regex: tasks\.(.*)
    target_label: job
    replacement: ${1}
    action: replace
- job_name: blackbox-tcp
  params:
    module:
    - tcp_connect
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /probe
  scheme: http
  dns_sd_configs:
  - names:
    - tasks.elasticsearch.service.acme
    refresh_interval: 30s
    type: A
    port: 9200
  - names:
    - tasks.logstash.service.acme
    refresh_interval: 30s
    type: A
    port: 5000
  relabel_configs:
  - source_labels: [__address__]
    separator: ;
    regex: (.*)(:80)?
    target_label: __param_target
    replacement: ${1}
    action: replace
  - source_labels: [__param_target]
    separator: ;
    regex: (.*)
    target_label: instance
    replacement: ${1}
    action: replace
  - source_labels: []
    separator: ;
    regex: .*
    target_label: __address__
    replacement: blackboxexporter:9115
    action: replace
  - source_labels: [__meta_dns_name]
    separator: ;
    regex: tasks\.(.*)
    target_label: job
    replacement: ${1}
    action: replace

• Alertmanager configuration file:

route:
  receiver: 'slack'
  repeat_interval: 3h #3h
  group_interval: 5m #5m
  group_wait: 1m #1m
  routes:
  #- receiver: 'logstash'
  #  continue: true
  - receiver: 'slack'

receivers:
  - name: 'slack'
    slack_configs:
      - send_resolved: true
        api_url: 'xxx'
        username: 'Prometheus - Alerts'
        channel: '#service-alerts'
        title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
        text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"
        icon_emoji: ':dart:'
  - name: 'logstash'
    webhook_configs:
      # Whether or not to notify about resolved alerts.
      - send_resolved: true
        # The endpoint to send HTTP POST requests to.
        url: 'http://logstash:8080/'

• Logs: