thanos: store/bucket: merging posting groups on 0.3.2-rc.0 segfault

Sometimes I get a segmentation fault. Seems related to the new posting group merging logic.

Thanos, Prometheus and Golang version used

improbable/thanos:v0.3.2-rc.0

What happened

Thanos Store crashed not long after starting up.

What you expected to happen

Thanos Store to work.

How to reproduce it (as minimally and precisely as possible):

Unfortunately I don’t have any reproducer. Perhaps something is visible from the stack trace.

Full logs to relevant components

Logs


level=debug ts=2019-02-28T09:32:11.084257964Z caller=bucket.go:660 msg="Blocks source resolutions" blocks=1 mint=1551338820000 maxt=1551346320000 lset="{monitor=\"prt
level=debug ts=2019-02-28T09:32:12.180438609Z caller=bucket.go:793 msg="series query processed" stats="&{blocksQueried:4 postingsTouched:20 postingsTouchedSizeSum:141
level=debug ts=2019-02-28T09:32:13.467076953Z caller=bucket.go:793 msg="series query processed" stats="&{blocksQueried:4 postingsTouched:10 postingsTouchedSizeSum:125
level=debug ts=2019-02-28T09:32:13.483760212Z caller=bucket.go:793 msg="series query processed" stats="&{blocksQueried:4 postingsTouched:10 postingsTouchedSizeSum:125
level=debug ts=2019-02-28T09:32:13.762218102Z caller=bucket.go:793 msg="series query processed" stats="&{blocksQueried:4 postingsTouched:20 postingsTouchedSizeSum:142
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x580e27]
goroutine 2851 [running]:
github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index.Merge(0xc44a3e4d00, 0x2, 0x2, 0x20, 0xc477dfa480)
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index/postings.go:375 +0x127
github.com/improbable-eng/thanos/pkg/store.merge(0xc44a3e4d00, 0x2, 0x2, 0xc477dfa480, 0xc5378ff340)
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:1246 +0x3f
github.com/improbable-eng/thanos/pkg/store.(*postingGroup).Postings(0xc4316cf000, 0xc4e45c9520, 0x2)
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:1242 +0x8c
github.com/improbable-eng/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings(0xc4316a04b0, 0xc44ab65f40, 0x5, 0x5, 0xbccaca661ee95d3e, 0xd82aef30d19f972c, 0xab0ed
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:1199 +0x200
github.com/improbable-eng/thanos/pkg/store.(*BucketStore).blockSeries(0xc4204e2090, 0x1203ce0, 0xc4316ce380, 0xda789a829d316901, 0x3f5ad73b1516dffa, 0xc4267b2480, 0xc
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:484 +0x8c
github.com/improbable-eng/thanos/pkg/store.(*BucketStore).Series.func1(0x433578, 0x1163da8)
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:706 +0xe6
github.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run.func1(0xc420f2b620, 0xc4316b0690, 0xc44a3bb660)
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:38 +0x27
created by github.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:37 +0xa8

Logs

panic: runtime error: invalid memory address or nil pointer dereference                                                                                               
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x580bc9]                                                                                              
goroutine 153969 [running]:                                                                                                                                          
github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index.(*intersectPostings).Seek(0xc55b47c0c0, 0x2354, 0xc55b47c0c0)                               
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index/postings.go:348 +0x29
github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index.(*intersectPostings).doNext(0xc55b47c0f0, 0x2354, 0x1050040)
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index/postings.go:323 +0x47                                       
github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index.(*intersectPostings).Next(0xc55b47c0f0, 0x5808ec)                                           
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index/postings.go:344 +0x60                                        
github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index.ExpandPostings(0x12047a0, 0xc55b47c0f0, 0x4, 0x12047a0, 0xc55b47c0f0, 0xc55c145a40, 0x2)    
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/tsdb/index/postings.go:250 +0x57                                       
github.com/improbable-eng/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings(0xc4646d8ff0, 0xc482c509f0, 0x3, 0x3, 0xfe11a8ab1c95568d, 0x4fd305f69ea32d8a, 0xff00d
b977420f744, 0x56a1fcc2250b3073, 0x0)                                                                                                                                                                               
/home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:1202 +0x2e3
github.com/improbable-eng/thanos/pkg/store.(*BucketStore).blockSeries(0xc420456000, 0x1203ce0, 0xc45ec2f180, 0x9b6e30349a296901, 0x47b2c82f01b614f8, 0xc4203f43f0, 0xc4646d8ff0, 0xc45ec997a0, 0xc482c509f0, 0x3, ...)                                                                                                                                                                      /home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:484 +0x8c                   
github.com/improbable-eng/thanos/pkg/store.(*BucketStore).Series.func1(0xc47b3a2000, 0x0)                                                                             /home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/pkg/store/bucket.go:706 +0xe6                                                                        
github.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run.func1(0xc5438b0480, 0xc428c1c7e0, 0xc5412b7540)
 /home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:38 +0x27                                                 
created by github.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run                                                                                  /home/bartek/Repos/thanosGo/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:37 +0xa8

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 2
Comments: 16 (16 by maintainers)

Most upvoted comments

We can repro it internally as well

EDIT: Now it’s gone. I think it’s tight to particular query & block. We will keep trying.

bwplotka on Mar 23, 2019