prometheus: Prometheus 1.2.0 suddenly stops scraping targets

What did you do? Since upgrading from version 1.1.3 to version 1.2.0, Prometheus stops scraping all targets (node, prometheus) after some time. After restarting prometheus, it works properly for some time, but then it stops scraping all targets again.

What did you expect to see? All targets should be in state “UP” and last scrape time should be less then 5 seconds (for node targets).

What did you see instead? Under which circumstances? Targets show “UNKNOWN” instead of “UP”.

Environment

• System information:

Linux 3.13.0-95-generic x86_64

• Prometheus version:

prometheus, version 1.2.0 (branch: master, revision: 522c93361459686fe3687f5ffe68c2ee34ea5c8e) build user: root@c8088ddaf2a8 build date: 20161007-12:53:55 go version: go1.6.3

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      hostname: 'my-hostname'

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  - "/etc/prometheus/rules/*.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']


  # Scrape the Node Exporter every 5 seconds.
  - job_name: 'node'
    scrape_interval: 5s
    file_sd_configs:
      - files:
        - /etc/prometheus/targets/*.yaml

    relabel_configs:
      - source_labels: [__address__]
        regex: (.*):9100
        replacement: ${1}
        target_label: instance
      - source_labels: [instance]
        regex: .*\.([\w,-]+)\.ct-app\.com
        replacement: ${1}
        target_label: region
      - source_labels: [instance]
        regex: (\w+).*
        replacement: ${1}
        target_label: customer
      - source_labels: [instance]
        regex: .*-(dev|stage|prod)-.*
        replacement: ${1}
        target_label: environment
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets: # Target to probe
        - www.domain.de
    relabel_configs:
      - source_labels: [__address__]
        regex: (.*)(:80)?
        target_label: __param_target
        replacement: ${1}
      - source_labels: [__param_target]
        regex: (.*)
        target_label: instance
        replacement: ${1}
      - source_labels: []
        regex: .*
        target_label: __address__
        replacement: blackbox:9115  # Blackbox exporter.