prometheus: panic: runtime error: invalid memory address or nil pointer dereference

What did you do?

What did you expect to see?

What did you see instead? Under which circumstances?

I see the panic in the title.

Environment

  • System information:

    Linux 2.6.32-358.el6.x86_64 x86_64

  • Prometheus version:

    2.0.0-beta.0, official build.

  • Alertmanager version:

    insert output of alertmanager -version here (if relevant to the issue)

  • Prometheus configuration file:

# my global config
global:
  scrape_interval:     30s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 30s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "first.rules.yml"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:

  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:39090']

#  - job_name: 'node'
#    static_configs:
#      - targets: ['localhost:39100']
#        labels:
#         group: 'prometheus-server'

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:39100']
        labels:
         group: 'prometheus-server'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: '4g'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: '4g'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: '4g'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: '4g'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: '4g'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: '4g'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: '4g'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: '4g'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: '4g'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: '4g'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: 'vp'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: '4g'

      - targets: ['8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100','8.8.8.8:39100']
        labels:
         group: '4g'


  - job_name: 'snmp_4g'
    scrape_interval:     3m
    scrape_timeout:     2m
    static_configs:
      - targets: ['8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8']
        labels:
         group: 'sw'
    metrics_path: /snmp
    params:
      module: [default]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 8.8.8.8:39116

  - job_name: 'snmp_4'
    scrape_interval:     3m
    scrape_timeout:     2m
    static_configs:
      - targets: ['8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8']
        labels:
         group: 'sw'
    metrics_path: /snmp
    params:
      module: [default]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 8.8.8.8:39116

  - job_name: 'snmp_4g'
    scrape_interval:     3m
    scrape_timeout:     2m
    static_configs:
      - targets: ['8.8.8.8']
        labels:
         group: 'sw'
    metrics_path: /snmp
    params:
      module: [default]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 8.8.8.8:39116

  - job_name: 'snmp'
    scrape_interval:     3m
    scrape_timeout:     2m
    static_configs:
      - targets: ['8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8']
        labels:
         group: 'sw'
      - targets: ['8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8']
        labels:
         group: 'sw'
      - targets: ['8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8']
        labels:
         group: 'swsnl'

    metrics_path: /snmp
    params:
      module: [default]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 8.8.8.8:39116

  - job_name: 'snmp'
    scrape_interval:     3m
    scrape_timeout:     2m
    static_configs:
      - targets: ['8.8.8.8','8.8.8.8']
        labels:
         group: 'sw'
    metrics_path: /snmp
    params:
      module: [default]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 8.8.8.8:39116

  - job_name: 'snmp'
    scrape_interval:     3m
    scrape_timeout:     2m
    static_configs:
      - targets: ['8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8']
        labels:
         group: 'sw'
    metrics_path: /snmp
    params:
      module: [default]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 8.8.8.8:39116

  - job_name: 'snmp'
    scrape_interval:     3m
    scrape_timeout:     2m
    static_configs:
      - targets: ['8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8','8.8.8.8']
        labels:
         group: 'sw'
    metrics_path: /snmp
    params:
      module: [default]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 8.8.8.8:39116

  - job_name: 'snmp'
    scrape_interval:     3m
    scrape_timeout:     2m
    static_configs:
      - targets: ['8.8.8.8','8.8.8.8']
        labels:
         group: 'sw'
    metrics_path: /snmp
    params:
      module: [default]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 8.8.8.8:39116

  - job_name: 'snmp'
    scrape_interval:     3m
    scrape_timeout:     2m
    static_configs:
      - targets: ['8.8.8.8','8.8.8.8','8.8.8.8']
        labels:
         group: 'sw'
    metrics_path: /snmp
    params:
      module: [default]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 8.8.8.8:39116
  • Alertmanager configuration file:
insert configuration here (if relevant to the issue)
  • Logs:
time="2017-07-14T15:45:59+08:00" level=info msg="Starting prometheus (version=2.0.0-beta.0, branch=master, revision=2b5d9159537cbd123219296121e05244e26c0940)" source="main.go:202" 
time="2017-07-14T15:45:59+08:00" level=info msg="Build context (go=go1.8.3, user=root@fc24486243df, date=20170712-12:21:13)" source="main.go:203" 
time="2017-07-14T15:45:59+08:00" level=info msg="Host details (Linux 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 ns1 (none))" source="main.go:204" 
time="2017-07-14T15:45:59+08:00" level=info msg="Starting tsdb" source="main.go:216" 
time="2017-07-14T15:48:01+08:00" level=info msg="tsdb started" source="main.go:222" 
time="2017-07-14T15:48:01+08:00" level=info msg="Loading configuration file prometheus.yml" source="main.go:351" 
time="2017-07-14T15:48:01+08:00" level=info msg="Listening on :39090" source="web.go:286" 
time="2017-07-14T15:48:01+08:00" level=info msg="Starting target manager..." source="targetmanager.go:67" 
ts=2017-07-14T07:48:31.805777219Z caller=compact.go:235 msg="compact blocks" blocks=[01BMZM1NMKB3862CZH451K3TKW]
ts=2017-07-14T08:00:00.158718549Z caller=db.go:867 msg="created head block" ulid=01BN01S43N9GVC7SPN7DC825SP mint=1500019200000 maxt=1500026400000
ts=2017-07-14T09:00:01.808299206Z caller=compact.go:235 msg="compact blocks" blocks=[01BMZTXCWNV88DXS2AW5VSKV3H]
ts=2017-07-14T09:00:12.039838082Z caller=compact.go:235 msg="compact blocks" blocks="[01BMZTXCZ04R8WFDAB6V56QPPA 01BN0143XXEDFBJP9GMQEF6902 01BN0571CG0AW8CFY0WMG4S3HT]"
ts=2017-07-14T10:00:00.156138675Z caller=db.go:867 msg="created head block" ulid=01BN08MVBN8W69Z1T5DKHC6FKE mint=1500026400000 maxt=1500033600000
ts=2017-07-14T11:00:01.818372645Z caller=compact.go:235 msg="compact blocks" blocks=[01BN01S43N9GVC7SPN7DC825SP]
ts=2017-07-14T12:00:00.154268008Z caller=db.go:867 msg="created head block" ulid=01BN0FGJKNTCM5V3R1BHG0DNP9 mint=1500033600000 maxt=1500040800000
ts=2017-07-14T13:00:01.81024889Z caller=compact.go:235 msg="compact blocks" blocks=[01BN08MVBN8W69Z1T5DKHC6FKE]
ts=2017-07-14T14:00:00.150149774Z caller=db.go:867 msg="created head block" ulid=01BN0PC9VMS07SFV674GKAQHKM mint=1500040800000 maxt=1500048000000
ts=2017-07-14T15:00:01.808408261Z caller=compact.go:235 msg="compact blocks" blocks=[01BN0FGJKNTCM5V3R1BHG0DNP9]
ts=2017-07-14T15:00:11.004250575Z caller=compact.go:235 msg="compact blocks" blocks="[01BN0C2RMTFXD4HQ58GXPJPD70 01BN0JYFWJGYWN2CRD7QMKE5X2 01BN0ST74GJPWC1YTY20F5BXRP]"
ts=2017-07-14T16:00:00.153046619Z caller=db.go:867 msg="created head block" ulid=01BN0X813NW0V7JF13AY3XRY6H mint=1500048000000 maxt=1500055200000
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1662f11]

goroutine 232 [running]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0xc420178620, 0xc45187e000, 0x1db8b, 0x20200, 0xed0fae9fa, 0x255cdb36, 0x29132a0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:759 +0x1a01
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc420178620, 0x29e8d60800, 0x1bf08eb000, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:624 +0x48c
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:281 +0x534
time="2017-07-15T23:10:01+08:00" level=info msg="Starting prometheus (version=2.0.0-beta.0, branch=master, revision=2b5d9159537cbd123219296121e05244e26c0940)" source="main.go:202" 
time="2017-07-15T23:10:01+08:00" level=info msg="Build context (go=go1.8.3, user=root@fc24486243df, date=20170712-12:21:13)" source="main.go:203" 
time="2017-07-15T23:10:01+08:00" level=info msg="Host details (Linux 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 ns1 (none))" source="main.go:204" 
time="2017-07-15T23:10:01+08:00" level=info msg="Starting tsdb" source="main.go:216" 
time="2017-07-15T23:11:29+08:00" level=info msg="tsdb started" source="main.go:222" 
time="2017-07-15T23:11:29+08:00" level=info msg="Loading configuration file prometheus.yml" source="main.go:351" 
time="2017-07-15T23:11:29+08:00" level=info msg="Starting target manager..." source="targetmanager.go:67" 
time="2017-07-15T23:11:29+08:00" level=info msg="Listening on :39090" source="web.go:286" 
ts=2017-07-15T15:11:30.134779123Z caller=db.go:867 msg="created head block" ulid=01BN3CVY9HY0796F9955C3E3Y0 mint=1500120000000 maxt=1500127200000
ts=2017-07-15T15:11:30.165463703Z caller=db.go:867 msg="created head block" ulid=01BN3CVYAPZAXS77ZDERSZRV1C mint=1500127200000 maxt=1500134400000
ts=2017-07-15T15:11:59.982241209Z caller=compact.go:235 msg="compact blocks" blocks=[01BN0PC9VMS07SFV674GKAQHKM]
ts=2017-07-15T15:12:08.962022735Z caller=compact.go:235 msg="compact blocks" blocks=[01BN0X813NW0V7JF13AY3XRY6H]
ts=2017-07-15T16:00:00.108907164Z caller=db.go:867 msg="created head block" ulid=01BN3FMR2100S2N0DN6MMZBKKK mint=1500134400000 maxt=1500141600000
ts=2017-07-15T17:00:29.984555746Z caller=compact.go:235 msg="compact blocks" blocks=[01BN3CVYAPZAXS77ZDERSZRV1C]
ts=2017-07-15T17:00:37.257511167Z caller=compact.go:235 msg="compact blocks" blocks="[01BN3CWVFE8JT4EA97M61WT31E 01BN3CX481ZGS0222QW9BEQMTW 01BN3K3GX0HS0AFHQY6PANB25S]"
ts=2017-07-15T18:00:00.108444905Z caller=db.go:867 msg="created head block" ulid=01BN3PGFA1F50EDHD6JAPZ30TN mint=1500141600000 maxt=1500148800000
ts=2017-07-15T19:00:29.986600081Z caller=compact.go:235 msg="compact blocks" blocks=[01BN3FMR2100S2N0DN6MMZBKKK]
ts=2017-07-15T20:00:00.11601485Z caller=db.go:867 msg="created head block" ulid=01BN3XC6J1MW0SMG7GHKX6TBAX mint=1500148800000 maxt=1500156000000
ts=2017-07-15T21:00:29.986676429Z caller=compact.go:235 msg="compact blocks" blocks=[01BN3PGFA1F50EDHD6JAPZ30TN]
ts=2017-07-15T22:00:00.1149678Z caller=db.go:867 msg="created head block" ulid=01BN447XT37BWGT9EJ5K2R1A7B mint=1500156000000 maxt=1500163200000
ts=2017-07-15T23:00:29.982945491Z caller=compact.go:235 msg="compact blocks" blocks=[01BN3XC6J1MW0SMG7GHKX6TBAX]
ts=2017-07-15T23:00:39.481496762Z caller=compact.go:235 msg="compact blocks" blocks="[01BN3SZ852H5ZSCK8PST8Q02FG 01BN40TZD2BM0QPYFZ2P05H5MR 01BN47PPMYFR32S28XKX57EVNS]"
ts=2017-07-16T00:00:00.111144418Z caller=db.go:867 msg="created head block" ulid=01BN4B3N227VMRWJDM7XJ0VXDV mint=1500163200000 maxt=1500170400000
ts=2017-07-16T01:00:29.973013395Z caller=compact.go:235 msg="compact blocks" blocks=[01BN447XT37BWGT9EJ5K2R1A7B]
ts=2017-07-16T02:00:00.112194233Z caller=db.go:867 msg="created head block" ulid=01BN4HZCA5GP7PSNVWZGXQE981 mint=1500170400000 maxt=1500177600000
ts=2017-07-16T03:00:29.985333474Z caller=compact.go:235 msg="compact blocks" blocks=[01BN4B3N227VMRWJDM7XJ0VXDV]
ts=2017-07-16T04:00:00.109957127Z caller=db.go:867 msg="created head block" ulid=01BN4RV3J36BWH4WQMPHBCKYH3 mint=1500177600000 maxt=1500184800000
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1662f11]

goroutine 276 [running]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0xc4201f21c0, 0xc434190000, 0x1db89, 0x20200, 0xed0fcde5a, 0x395ec127, 0x29132a0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:759 +0x1a01
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc4201f21c0, 0x29e8d60800, 0x1bf08eb000, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:624 +0x48c
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:281 +0x534

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 31 (11 by maintainers)

Commits related to this issue

Most upvoted comments

@tudor and I traced this down to app.Add() that returns duplicate refs for different metrics here: https://github.com/prometheus/prometheus/blob/dev-2.0/retrieval/scrape.go#L833. When there are duplicate refs two elements in c.refs points to the same element in c.lsets. This means that a call to addRef leaves one element in c.refs without its corresponding c.lsets element because it’s deleted here: https://github.com/prometheus/prometheus/blob/dev-2.0/retrieval/scrape.go#L549

Here is a sample of our logging that we added:

time="2017-09-05T20:14:02Z" level=info msg="Not found in the cache, producing ref 015e53a12d914b5a37007a6fd8f30b73000000000002083a for metric container_cpu_cfs_periods_total{id="/kubepods/burstable/pod2f30f3e5-8844-11e7-94df-0a365eaab0be"}" source="scrape.go:837" target="{__address__="XXXX:443", __metrics_path__="/api/v1/nodes/ip-10-0-114-253.ec2.internal:4194/proxy/metrics", __scheme__="https", beta_kubernetes_io_arch="amd64", beta_kubernetes_io_instance_type="m4.large", beta_kubernetes_io_os="linux", failure_domain_beta_kubernetes_io_region="us-east-1", failure_domain_beta_kubernetes_io_zone="us-east-1c", instance="ip-10-0-114-253.ec2.internal", job="kubernetes-cadvisor", kubernetes_io_hostname="ip-10-0-114-253.ec2.internal", kubernetes_io_role="master", node_role_kubernetes_io_master=""}"
time="2017-09-05T20:14:02Z" level=info msg="Not found in the cache, producing ref 015e53a12d914b5a37007a6fd8f30b73000000000002083a for metric container_cpu_cfs_periods_total{id="/kubepods/pod6492c4a0-8d1d-11e7-a32b-12a0b9c354ec"}" source="scrape.go:837" target="{__address__="XXXX:443", __metrics_path__="/api/v1/nodes/ip-10-0-114-253.ec2.internal:4194/proxy/metrics", __scheme__="https", beta_kubernetes_io_arch="amd64", beta_kubernetes_io_instance_type="m4.large", beta_kubernetes_io_os="linux", failure_domain_beta_kubernetes_io_region="us-east-1", failure_domain_beta_kubernetes_io_zone="us-east-1c", instance="ip-10-0-114-253.ec2.internal", job="kubernetes-cadvisor", kubernetes_io_hostname="ip-10-0-114-253.ec2.internal", kubernetes_io_role="master", node_role_kubernetes_io_master=""}"
panic: Duplicate ref!

As you can see both container_cpu_cfs_periods_total{id="/kubepods/burstable/pod2f30f3e5-8844-11e7-94df-0a365eaab0be"}" and container_cpu_cfs_periods_total{id="/kubepods/pod6492c4a0-8d1d-11e7-a32b-12a0b9c354ec"}" maps to the same ref 015e53a12d914b5a37007a6fd8f30b73000000000002083a.

We also looked at where the uniqueness of refs is guaranteed, it seems to be from the length of h.series here: https://github.com/prometheus/tsdb/blob/master/head.go#L709. This sounds like a race issue, but we weren’t able to uncover it by reading the code.

+4
igorcanadi on Sep 5, 2017

Similar issue (not sure if same). Nothing special in config, static targets + remote write to remote storage adapter.

prometheus, version 2.0.0-beta.2 (branch: HEAD, revision: a52f082939a566d5269671e98be06fc6bdf61d09)

[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x166e410]

goroutine 364 [running]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0xc420207180, 0xc566666000, 0xeb58c, 0x101e00, 0xed12df87b, 0x341233e8, 0x294c1e0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:777 +0x19a0
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc420207180, 0x37e11d600, 0x37e11d600, 0x0)
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:646 +0x48c
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:302 +0x534
2017-08-22 10:03:41.996 - INFO - Process exited with code 2
+2
bkupidura on Aug 22, 2017
Read more comments on GitHub
← prometheus: opening storage failed: found unsequential head chunk files 23 and 25
prometheus: Possible race condition in TSDB head chunks →