influxdb: influxdb memory increase quickly after query and quickly OOM crashed.

server info

System info: InfluxDB version 1.4.2, Linux version 4.4.19-29.55.amzn1.x86_64 (mockbuild@gobi-build-64012) , with free memory about 12G.

influxdb config

influxdb.conf.txt

Steps to reproduce:

i create a measurement service_access, and it keep a increasement about 1000/s . then i use the the query commond ‘SELECT * FROM server_monitor.autogen.service_access ORDER BY time DESC LIMIT 10000’.

every time i execute the query sql, the free memory reduce quickly , with a speed of 500m per query. after about execute 20 times of the query sql, the influxdb crashed with out of memoery error.

Expected behavior: The influxdb run normally

Actual behavior: influxdb use too much memory, and crashed with OOM exception.

LOG File :

`[httpd] 172.31.30.11,172.31.2.243 - root [09/Jan/2018:09:52:10 +0800] “POST /write?consistency=one&db=server_monitor&p=%5BREDACTED%5D&precision=n&rp=autogen&u=root HTTP/1.1” 204 0 “-” “okhttp/2.4.0” b502ed05-f4df-11e7-8087-000000000000 16135 [httpd] 172.31.30.10,172.31.1.140 - root [09/Jan/2018:09:52:29 +0800] “POST /write?consistency=one&db=server_monitor&p=%5BREDACTED%5D&precision=n&rp=autogen&u=root HTTP/1.1” 204 0 “-” “okhttp/2.4.0” c0478a52-f4df-11e7-8088-000000000000 15735 [I] 2018-01-09T01:52:35Z SELECT * FROM server_monitor.autogen.service_access ORDER BY time DESC LIMIT 10000 service=query [httpd] 172.31.30.12,172.31.2.243 - root [09/Jan/2018:09:52:35 +0800] “POST /write?consistency=one&db=server_monitor&p=%5BREDACTED%5D&precision=n&rp=autogen&u=root HTTP/1.1” 204 0 “-” “okhttp/2.4.0” c3e2d47b-f4df-11e7-808a-000000000000 737548 [httpd] 127.0.0.1 - - [09/Jan/2018:09:52:35 +0800] “POST /query?chunked=true&db=server_monitor&epoch=ns&q=select+%2A+from+service_access+order+by+time+desc+limit+10000%3B HTTP/1.1” 200 310777 “-” “InfluxDBShell/1.4.2” c3beddea-f4df-11e7-8089-000000000000 1418485 [httpd] 172.31.30.13,172.31.1.140 - root [09/Jan/2018:09:52:39 +0800] “POST /write?consistency=one&db=server_monitor&p=%5BREDACTED%5D&precision=n&rp=autogen&u=root HTTP/1.1” 204 0 “-” “okhttp/2.4.0” c5e54d9c-f4df-11e7-808b-000000000000 25676 [httpd] 172.31.30.11,172.31.2.243 - root [09/Jan/2018:09:52:40 +0800] “POST /write?consistency=one&db=server_monitor&p=%5BREDACTED%5D&precision=n&rp=autogen&u=root HTTP/1.1” 204 0 “-” “okhttp/2.4.0” c6aba55a-f4df-11e7-808c-000000000000 35117 [I] 2018-01-09T01:52:46Z SELECT * FROM server_monitor.autogen.service_access ORDER BY time DESC LIMIT 10000 service=query fatal error: runtime: out of memory

runtime stack: runtime.throw(0xcc89cf, 0x16) /usr/local/go/src/runtime/panic.go:605 +0x95 runtime.sysMap(0xc6ba020000, 0x4cbf0000, 0x1b000336800, 0x114ad98) /usr/local/go/src/runtime/mem_linux.go:216 +0x1d0 runtime.(*mheap).sysAlloc(0x11318c0, 0x4cbf0000, 0x7f0c45b97c90) /usr/local/go/src/runtime/malloc.go:470 +0xd7 runtime.(*mheap).grow(0x11318c0, 0x265f2, 0x0) /usr/local/go/src/runtime/mheap.go:887 +0x60 runtime.(*mheap).allocSpanLocked(0x11318c0, 0x265f2, 0x114ada8, 0x7f0c45981d18) /usr/local/go/src/runtime/mheap.go:800 +0x334 runtime.(*mheap).alloc_m(0x11318c0, 0x265f2, 0x7f0c57ff0101, 0x416bbc) /usr/local/go/src/runtime/mheap.go:666 +0x118 runtime.(*mheap).alloc.func1() /usr/local/go/src/runtime/mheap.go:733 +0x4d runtime.systemstack(0x7f0c57ffedd0) /usr/local/go/src/runtime/asm_amd64.s:360 +0xab runtime.(*mheap).alloc(0x11318c0, 0x265f2, 0x7f0c57010101, 0x416224) /usr/local/go/src/runtime/mheap.go:732 +0xa1 runtime.largeAlloc(0x4cbe2166, 0x7f0c67c20101, 0xc5cc674076) /usr/local/go/src/runtime/malloc.go:827 +0x98 runtime.mallocgc.func1() /usr/local/go/src/runtime/malloc.go:722 +0x46 runtime.systemstack(0xc42001ec00) /usr/local/go/src/runtime/asm_amd64.s:344 +0x79 runtime.mstart() /usr/local/go/src/runtime/proc.go:1135

goroutine 10699 [running]: runtime.systemstack_switch() /usr/local/go/src/runtime/asm_amd64.s:298 fp=0xc420420530 sp=0xc420420528 pc=0x45b710 runtime.mallocgc(0x4cbe2166, 0xb8bac0, 0x4cbe2101, 0x5) /usr/local/go/src/runtime/malloc.go:721 +0x7ae fp=0xc4204205d8 sp=0xc420420530 pc=0x41289e runtime.makeslice(0xb8bac0, 0x4cbe2166, 0x4cbe2166, 0x4cbe2166, 0x5, 0x0) /usr/local/go/src/runtime/slice.go:54 +0x77 fp=0xc420420608 sp=0xc4204205d8 pc=0x444f77 github.com/golang/snappy.Decode(0x0, 0x0, 0x0, 0x7f0c2e5820d8, 0x5902261, 0x11a7d70d, 0x4488, 0xc5dac0, 0xc595be6601, 0xc4204206d0, …) /go/src/github.com/golang/snappy/decode.go:63 +0x1f4 fp=0xc420420658 sp=0xc420420608 pc=0x91bb74 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*StringDecoder).SetBytes(0xc42172c800, 0x7f0c2e5820d7, 0x5902262, 0x11a7d70e, 0xc420420101, 0xc420420740) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/string.go:80 +0x12c fp=0xc4204206e0 sp=0xc420420658 pc=0xad690c github.com/influxdata/influxdb/tsdb/engine/tsm1.DecodeStringBlock.func1(0xc4201ac000, 0x7f0c2e580a0e, 0x16c9, 0x11a7edd7, 0xc420420810, 0xc42172c800, 0x7f0c2e5820d7, 0x5902262, 0x11a7d70e, 0xc4204207e8, …) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/encoding.go:931 +0x8b fp=0xc420420750 sp=0xc4204206e0 pc=0xaea5ab github.com/influxdata/influxdb/tsdb/engine/tsm1.DecodeStringBlock(0x7f0c2e580a0b, 0x590392e, 0x11a7edda, 0xc59591fd30, 0x8, 0x8, 0xc59546d088, 0x7f0c67aace18, 0x0) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/encoding.go:955 +0x24d fp=0xc420420850 sp=0xc420420750 pc=0xa9ae3d github.com/influxdata/influxdb/tsdb/engine/tsm1.(*mmapAccessor).readStringBlock(0xc4203ac9b0, 0xc595bf6b10, 0xc59591fd30, 0x0, 0x0, 0xc59546d088, 0xc59593afc0, 0x20) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:1352 +0xc3 fp=0xc4204208d8 sp=0xc420420850 pc=0xad3383 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*TSMReader).ReadStringBlockAt(0xc420333560, 0xc595bf6b10, 0xc59591fd30, 0x1, 0xc59546d088, 0x0, 0x1, 0x0) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:330 +0x71 fp=0xc420420958 sp=0xc4204208d8 pc=0xacd6d1 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*KeyCursor).ReadStringBlock(0xc595bf2990, 0xc59591fd30, 0x0, 0x0, 0xc4200140c0, 0xc4203503c0, 0xc595be66c0) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.gen.go:550 +0x88 fp=0xc420420a90 sp=0xc420420958 pc=0xab4b58 github.com/influxdata/influxdb/tsdb/engine/tsm1.newStringDescendingCursor(0x7ffffffffffffffe, 0x0, 0x0, 0x0, 0xc595bf2990, 0xc595bf2990) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:1886 +0x116 fp=0xc420420b18 sp=0xc420420a90 pc=0xac7b16 github.com/influxdata/influxdb/tsdb/engine/tsm1.newStringCursor(0x7ffffffffffffffe, 0x10e2400, 0x0, 0x0, 0x0, 0xc595bf2990, 0x7ffffffffffffffe, 0xc59591fc00) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/iterator.gen.go:1747 +0xb6 fp=0xc420420b58 sp=0xc420420b18 pc=0xac72b6 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).buildStringCursor(0xc42044a7e0, 0x10e24e0, 0xc4200140c0, 0xc584abd0d0, 0xe, 0xc42146f980, 0x7a, 0xc4212237f5, 0x5, 0x0, …) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/batch_cursor.gen.go:70 +0x2d6 fp=0xc420420df0 sp=0xc420420b58 pc=0xa70886 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).buildCursor(0xc42044a7e0, 0x10e24e0, 0xc4200140c0, 0xc584abd0d0, 0xe, 0xc42146f980, 0x7a, 0xc595bf2750, 0x3, 0x3, …) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2256 +0xd5c fp=0xc420421020 sp=0xc420420df0 pc=0xaae9cc github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createVarRefSeriesIterator(0xc42044a7e0, 0x10e24e0, 0xc4200140c0, 0x0, 0xc584abd0d0, 0xe, 0xc42146f980, 0x7a, 0xc48f8d6eb0, 0x0, …) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2041 +0x2c35 fp=0xc420421b88 sp=0xc420421020 pc=0xaad655 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createTagSetGroupIterators(0xc42044a7e0, 0x10e24e0, 0xc4200140c0, 0x0, 0xc584abd0d0, 0xe, 0xc5a46d1660, 0x122, 0x69a, 0xc48f8d6eb0, …) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1978 +0x1e3 fp=0xc420421dc0 sp=0xc420421b88 pc=0xaaa5a3 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createTagSetIterators.func1(0xc584abd2d0, 0xc575d4eb00, 0x8, 0x8, 0xc42044a7e0, 0x10e24e0, 0xc4200140c0, 0x0, 0xc584abd0d0, 0xe, …) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1938 +0x146 fp=0xc420421f78 sp=0xc420421dc0 pc=0xaed8a6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420421f80 sp=0xc420421f78 pc=0x45e341 created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createTagSetIterators /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1936 +0x53a

goroutine 1 [chan receive, 13 minutes]: main.(*Main).Run(0xc4201625c0, 0xc420010150, 0x4, 0x4, 0x0, 0x11296e0) /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:98 +0x3c1 main.main() /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:46 +0xef

goroutine 5 [syscall, 13 minutes]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:131 +0xa6 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:22 +0x22 created by os/signal.init.0 /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 18 [IO wait, 13 minutes]: internal/poll.runtime_pollWait(0x7f0c67bc7f70, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc420164b98, 0x72, 0xc420026500, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc420164b98, 0xffffffffffffff00, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Accept(0xc420164b80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e2 net.(*netFD).accept(0xc420164b80, 0x0, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc42015c220, 0xc4200266e8, 0x47542e, 0xc42017df20) /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e net.(*TCPListener).Accept(0xc42015c220, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/tcpsock.go:247 +0x49 github.com/influxdata/influxdb/tcp.(*Mux).Serve(0xc42017df20, 0x10e0e20, 0xc42015c220, 0xc42015c220, 0x0) /go/src/github.com/influxdata/influxdb/tcp/mux.go:75 +0x97 created by github.com/influxdata/influxdb/cmd/influxd/run.(*Server).Open /go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:374 +0x248

goroutine 136 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc420462120, 0xc42042d560) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1281 +0x263 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc420462120, 0xc42042d560) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:267 +0x61 created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:267 +0x11f

goroutine 135 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc420462120, 0xc42042d440) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1241 +0x129 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc420462120, 0xc42042d440) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:317 +0x61 created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:317 +0x10b

goroutine 137 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc42044a7e0, 0xc42042d740) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1241 +0x129 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc42044a7e0, 0xc42042d740) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:317 +0x61 created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:317 +0x10b

goroutine 138 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc42044a7e0, 0xc42042d860) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1281 +0x263 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc42044a7e0, 0xc42042d860) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:267 +0x61 created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:267 +0x11f

goroutine 153 [select]: github.com/influxdata/influxdb/tsdb.(*Store).monitorShards(0xc42021b1e0) /go/src/github.com/influxdata/influxdb/tsdb/store.go:1451 +0x214 created by github.com/influxdata/influxdb/tsdb.(*Store).Open /go/src/github.com/influxdata/influxdb/tsdb/store.go:156 +0x292

goroutine 154 [select]: github.com/influxdata/influxdb/services/subscriber.(*Service).run(0xc4201703c0) /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:238 +0x313 github.com/influxdata/influxdb/services/subscriber.(*Service).Open.func1(0xc4201703c0) /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:97 +0x57 created by github.com/influxdata/influxdb/services/subscriber.(*Service).Open /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:95 +0x178

goroutine 155 [select, 13 minutes]: github.com/influxdata/influxdb/services/subscriber.(*Service).waitForMetaUpdates(0xc4201703c0) /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:164 +0xff github.com/influxdata/influxdb/services/subscriber.(*Service).Open.func2(0xc4201703c0) /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:101 +0x57 created by github.com/influxdata/influxdb/services/subscriber.(*Service).Open /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:99 +0x19a

goroutine 156 [select]: github.com/influxdata/influxdb/monitor.(*Monitor).storeStatistics(0xc420204000) /go/src/github.com/influxdata/influxdb/monitor/service.go:428 +0x6be created by github.com/influxdata/influxdb/monitor.(*Monitor).Open /go/src/github.com/influxdata/influxdb/monitor/service.go:125 +0x36a

goroutine 157 [select, 3 minutes]: github.com/influxdata/influxdb/services/precreator.(*Service).runPrecreation(0xc42015abe0) /go/src/github.com/influxdata/influxdb/services/precreator/service.go:77 +0x14c created by github.com/influxdata/influxdb/services/precreator.(*Service).Open /go/src/github.com/influxdata/influxdb/services/precreator/service.go:55 +0x19f

goroutine 158 [chan receive, 13 minutes]: github.com/influxdata/influxdb/tcp.(*listener).Accept(0xc4202062b0, 0xceac00, 0xc420224120, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/tcp/mux.go:199 +0x4e github.com/influxdata/influxdb/services/snapshotter.(*Service).serve(0xc420224120) /go/src/github.com/influxdata/influxdb/services/snapshotter/service.go:88 +0x78 created by github.com/influxdata/influxdb/services/snapshotter.(*Service).Open /go/src/github.com/influxdata/influxdb/services/snapshotter/service.go:61 +0xa1

goroutine 159 [select]: github.com/influxdata/influxdb/services/continuous_querier.(*Service).backgroundLoop(0xc4201fa240) /go/src/github.com/influxdata/influxdb/services/continuous_querier/service.go:219 +0x1de created by github.com/influxdata/influxdb/services/continuous_querier.(*Service).Open /go/src/github.com/influxdata/influxdb/services/continuous_querier/service.go:132 +0x1af

goroutine 160 [IO wait]: internal/poll.runtime_pollWait(0x7f0c67bc7df0, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc4203a4118, 0x72, 0xc4216fec00, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc4203a4118, 0xffffffffffffff00, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Accept(0xc4203a4100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e2 net.(*netFD).accept(0xc4203a4100, 0xc4216fedf0, 0x401c27, 0xc4210c1200) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc42000e0c8, 0x68f548, 0x45aa80, 0xc4216fee38) /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e net.(*TCPListener).Accept(0xc42000e0c8, 0xcea488, 0xc4210c1180, 0x10e2560, 0xc4218e5aa0) /usr/local/go/src/net/tcpsock.go:247 +0x49 net/http.(*Server).Serve(0xc42066dba0, 0x10e0e20, 0xc42000e0c8, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2695 +0x1b2 net/http.Serve(0x10e0e20, 0xc42000e0c8, 0x10d38e0, 0xc4201769a0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2323 +0x73 github.com/influxdata/influxdb/services/httpd.(*Service).serve(0xc42017ed20, 0x10e0e20, 0xc42000e0c8) /go/src/github.com/influxdata/influxdb/services/httpd/service.go:221 +0x62 github.com/influxdata/influxdb/services/httpd.(*Service).serveTCP(0xc42017ed20) /go/src/github.com/influxdata/influxdb/services/httpd/service.go:209 +0x3c created by github.com/influxdata/influxdb/services/httpd.(*Service).Open /go/src/github.com/influxdata/influxdb/services/httpd/service.go:160 +0x5d6

goroutine 161 [select, 13 minutes]: github.com/influxdata/influxdb/services/retention.(*Service).run(0xc420224480) /go/src/github.com/influxdata/influxdb/services/retention/service.go:77 +0xcd7 github.com/influxdata/influxdb/services/retention.(*Service).Open.func1(0xc420224480) /go/src/github.com/influxdata/influxdb/services/retention/service.go:50 +0x57 created by github.com/influxdata/influxdb/services/retention.(*Service).Open /go/src/github.com/influxdata/influxdb/services/retention/service.go:50 +0x194

goroutine 162 [select, 13 minutes]: github.com/influxdata/influxdb/cmd/influxd/run.(*Server).startServerReporting(0xc4201803c0) /go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:503 +0x14f created by github.com/influxdata/influxdb/cmd/influxd/run.(*Server).Open /go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:448 +0xe5e

goroutine 163 [select, 13 minutes]: github.com/influxdata/influxdb/cmd/influxd/run.(*Command).monitorServerErrors(0xc4201768f0) /go/src/github.com/influxdata/influxdb/cmd/influxd/run/command.go:148 +0x1cf created by github.com/influxdata/influxdb/cmd/influxd/run.(*Command).Run /go/src/github.com/influxdata/influxdb/cmd/influxd/run/command.go:129 +0xa5e

goroutine 164 [select, 13 minutes, locked to thread]: runtime.gopark(0xceaac0, 0x0, 0xcbc691, 0x6, 0x18, 0x1) /usr/local/go/src/runtime/proc.go:287 +0x12c runtime.selectgo(0xc42027df50, 0xc420f7bd40) /usr/local/go/src/runtime/select.go:395 +0x1149 runtime.ensureSigM.func1() /usr/local/go/src/runtime/signal_unix.go:511 +0x220 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 165 [IO wait]: internal/poll.runtime_pollWait(0x7f0c67bc7eb0, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc4203a4198, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc4203a4198, 0xc420926000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc4203a4180, 0xc420926000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc4203a4180, 0xc420926000, 0x1000, 0x1000, 0xc420425b00, 0x68584a, 0xc4214c8518) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc42000e0d0, 0xc420926000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc4214c8510, 0xc420926000, 0x1000, 0x1000, 0xc420425be8, 0x59aa0c, 0xc4203a4180) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc420509b60) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).Peek(0xc420509b60, 0x4, 0x0, 0x0, 0x0, 0x0, 0x7f0c67bc7e00) /usr/local/go/src/bufio/bufio.go:129 +0x3a net/http.(*conn).readRequest(0xc42017e000, 0x10e24a0, 0xc421446ec0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:930 +0xc77 net/http.(*conn).serve(0xc42017e000, 0x10e24a0, 0xc421446ec0) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 37 [IO wait]: internal/poll.runtime_pollWait(0x7f0c67bc7c70, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc420480118, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc420480118, 0xc421c8a000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc420480100, 0xc421c8a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc420480100, 0xc421c8a000, 0x1000, 0x1000, 0xc42041fb00, 0x68584a, 0xc421c7e0c8) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc420464008, 0xc421c8a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc421c7e0c0, 0xc421c8a000, 0x1000, 0x1000, 0xc42041fbe8, 0x59aa0c, 0xc420480100) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc42148a180) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).Peek(0xc42148a180, 0x4, 0x0, 0x0, 0x0, 0x0, 0x7f0c67bc7c00) /usr/local/go/src/bufio/bufio.go:129 +0x3a net/http.(*conn).readRequest(0xc421c80000, 0x10e24a0, 0xc42079f7c0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:930 +0xc77 net/http.(*conn).serve(0xc421c80000, 0x10e24a0, 0xc42079f7c0) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10619 [IO wait, 1 minutes]: internal/poll.runtime_pollWait(0x7f0c67bc7af0, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc57b9dc998, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc57b9dc998, 0xc422efb000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc57b9dc980, 0xc422efb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc57b9dc980, 0xc422efb000, 0x1000, 0x1000, 0xc4216b78c8, 0x68587b, 0xc4c240) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc4222765d8, 0xc422efb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc51a4e5710, 0xc422efb000, 0x1000, 0x1000, 0x18, 0xc4216b7a10, 0x406313) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc42246ed80) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc42246ed80, 0x112040a, 0xc42017c4d8, 0x0, 0x0, 0xf2, 0x32) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc42246ed80, 0x100, 0xf8, 0xca9c60, 0xc4216b7a98, 0x220000402db9, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc51a4e54d0, 0xc4216b7b20, 0xc4216b7b20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc51a4e54d0, 0xc57b9e6500, 0x0, 0xc4216b7b90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc42246ed80, 0x0, 0xc57b9e6500, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc58d18df40, 0x10e24a0, 0xc57b9e2740, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc58d18df40, 0x10e24a0, 0xc57b9e2740) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 251 [IO wait, 11 minutes]: internal/poll.runtime_pollWait(0x7f0c67bc78b0, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc420458598, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc420458598, 0xc422f45000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc420458580, 0xc422f45000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc420458580, 0xc422f45000, 0x1000, 0x1000, 0xc4225118c8, 0x68584a, 0xc4214c8788) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc4204641a0, 0xc422f45000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc4214c8780, 0xc422f45000, 0x1000, 0x1000, 0xc420100240, 0x3e0000000003, 0x398) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc4201dcc00) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc4201dcc00, 0xc422511a0a, 0x411f36, 0x7f0c640474f0, 0x1, 0xc422511ab8, 0x4123c7) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc4201dcc00, 0x100, 0xf8, 0xca9c60, 0x1, 0x122002002e700, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc422808060, 0xc422511b20, 0xc422511b20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc422808060, 0xc4214c4100, 0x0, 0xc422511b90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc4201dcc00, 0x0, 0xc4214c4100, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc4210c0d20, 0x10e24a0, 0xc42172c240, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc4210c0d20, 0x10e24a0, 0xc42172c240) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 278 [IO wait, 11 minutes]: internal/poll.runtime_pollWait(0x7f0c67bc77f0, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc420314098, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc420314098, 0xc420a6a000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc420314080, 0xc420a6a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc420314080, 0xc420a6a000, 0x1000, 0x1000, 0xc42250d8c8, 0x68584a, 0xc4227d4098) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc422276018, 0xc420a6a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc4227d4090, 0xc420a6a000, 0x1000, 0x1000, 0xc42044c000, 0xc42250d9a0, 0xc4207da700) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc4241e6180) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc4241e6180, 0xc42250da0a, 0x411f36, 0x7f0c67baad08, 0x1, 0xc42250dab8, 0x4123c7) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc4241e6180, 0x100, 0xf8, 0xca9c60, 0x1, 0x1220020146000, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc4212ca360, 0xc42250db20, 0xc42250db20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc4212ca360, 0xc4214c2100, 0x0, 0xc42250db90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc4241e6180, 0x0, 0xc4214c2100, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc421370a00, 0x10e24a0, 0xc4228ac200, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc421370a00, 0x10e24a0, 0xc4228ac200) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10610 [IO wait, 1 minutes]: internal/poll.runtime_pollWait(0x7f0c67bc7730, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc5b3127b98, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc5b3127b98, 0xc424640000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc5b3127b80, 0xc424640000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc5b3127b80, 0xc424640000, 0x1000, 0x1000, 0xc4216958c8, 0x68587b, 0xc4c240) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc42000ff20, 0xc424640000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc4214c9d70, 0xc424640000, 0x1000, 0x1000, 0x18, 0xc421695a10, 0x406313) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc51a0f7ec0) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc51a0f7ec0, 0xc422d9670a, 0xc42017c4d8, 0x0, 0x0, 0xf2, 0x32) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc51a0f7ec0, 0x100, 0xf8, 0xca9c60, 0xc421695a98, 0x220000402db9, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc4214c9a10, 0xc421695b20, 0xc421695b20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc4214c9a10, 0xc494823c00, 0x0, 0xc421695b90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc51a0f7ec0, 0x0, 0xc494823c00, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc49274df40, 0x10e24a0, 0xc5a6d77dc0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc49274df40, 0x10e24a0, 0xc5a6d77dc0) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10674 [IO wait]: internal/poll.runtime_pollWait(0x7f0c67bc71f0, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc56a6ea898, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc56a6ea898, 0xc42241d000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc56a6ea880, 0xc42241d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc56a6ea880, 0xc42241d000, 0x1000, 0x1000, 0xc4216f88c8, 0x68587b, 0xc4c240) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc44fbcdba0, 0xc42241d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc5c7550c90, 0xc42241d000, 0x1000, 0x1000, 0x18, 0xc4216f8a10, 0x406313) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc5726f12c0) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc5726f12c0, 0xc420b4450a, 0xc42017c4d8, 0x0, 0x0, 0xc4216f8ab8, 0x4123c7) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc5726f12c0, 0x100, 0xf8, 0xca9c60, 0x8, 0x220020316e00, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc5c7550cc0, 0xc4216f8b20, 0xc4216f8b20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc5c7550cc0, 0xc5a3536300, 0x0, 0xc4216f8b90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc5726f12c0, 0x0, 0xc5a3536300, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc4210c10e0, 0x10e24a0, 0xc51a876380, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc4210c10e0, 0x10e24a0, 0xc51a876380) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10702 [running]: goroutine running on other thread; stack unavailable created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createTagSetIterators /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1936 +0x53a

goroutine 6066 [chan receive]: github.com/influxdata/influxdb/services/httpd.(*Handler).serveQuery(0xc4201769a0, 0x7f0c67b47bc0, 0xc5846cf720, 0xc43ef12000, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:475 +0x896 github.com/influxdata/influxdb/services/httpd.(*Handler).(github.com/influxdata/influxdb/services/httpd.serveQuery)-fm(0x7f0c67b47bc0, 0xc5846cf720, 0xc43ef12000, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:135 +0x5c github.com/influxdata/influxdb/services/httpd.authenticate.func1(0x7f0c67b47bc0, 0xc5846cf720, 0xc43ef12000) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1332 +0x7da net/http.HandlerFunc.ServeHTTP(0xc42015fdc0, 0x7f0c67b47bc0, 0xc5846cf720, 0xc43ef12000) /usr/local/go/src/net/http/server.go:1918 +0x44 github.com/influxdata/influxdb/services/httpd.(*Handler).responseWriter.func1(0x10de920, 0xc48f8d6c80, 0xc43ef12000) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1509 +0xab net/http.HandlerFunc.ServeHTTP(0xc42015fde0, 0x10de920, 0xc48f8d6c80, 0xc43ef12000) /usr/local/go/src/net/http/server.go:1918 +0x44 github.com/influxdata/influxdb/services/httpd.gzipFilter.func1(0x10de9a0, 0xc5846cf700, 0xc43ef12000) /go/src/github.com/influxdata/influxdb/services/httpd/gzip.go:39 +0x215 net/http.HandlerFunc.ServeHTTP(0xc42015fe00, 0x10de9a0, 0xc5846cf700, 0xc43ef12000) /usr/local/go/src/net/http/server.go:1918 +0x44 github.com/influxdata/influxdb/services/httpd.cors.func1(0x10de9a0, 0xc5846cf700, 0xc43ef12000) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1454 +0x102 net/http.HandlerFunc.ServeHTTP(0xc42015fe20, 0x10de9a0, 0xc5846cf700, 0xc43ef12000) /usr/local/go/src/net/http/server.go:1918 +0x44 github.com/influxdata/influxdb/services/httpd.requestID.func1(0x10de9a0, 0xc5846cf700, 0xc43ef12000) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1485 +0x179 net/http.HandlerFunc.ServeHTTP(0xc42015fe40, 0x10de9a0, 0xc5846cf700, 0xc43ef12000) /usr/local/go/src/net/http/server.go:1918 +0x44 github.com/influxdata/influxdb/services/httpd.(*Handler).logging.func1(0x10de9a0, 0xc5846cf6e0, 0xc43ef12000) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1493 +0xe1 net/http.HandlerFunc.ServeHTTP(0xc42015fe60, 0x10de9a0, 0xc5846cf6e0, 0xc43ef12000) /usr/local/go/src/net/http/server.go:1918 +0x44 github.com/influxdata/influxdb/services/httpd.(*Handler).recovery.func1(0x10e10a0, 0xc5159b90a0, 0xc43ef12000) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:1546 +0x14c net/http.HandlerFunc.ServeHTTP(0xc42015fe80, 0x10e10a0, 0xc5159b90a0, 0xc43ef12000) /usr/local/go/src/net/http/server.go:1918 +0x44 github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc42015fbc0, 0x10e10a0, 0xc5159b90a0, 0xc43ef12000) /go/src/github.com/bmizerany/pat/mux.go:117 +0x15b github.com/influxdata/influxdb/services/httpd.(*Handler).ServeHTTP(0xc4201769a0, 0x10e10a0, 0xc5159b90a0, 0xc43ef12000) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:283 +0x24d net/http.serverHandler.ServeHTTP(0xc42066dba0, 0x10e10a0, 0xc5159b90a0, 0xc43ef12000) /usr/local/go/src/net/http/server.go:2619 +0xb4 net/http.(*conn).serve(0xc421370b40, 0x10e24a0, 0xc42227c840) /usr/local/go/src/net/http/server.go:1801 +0x71d created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10693 [select]: github.com/influxdata/influxdb/services/httpd.(*Handler).serveQuery.func2(0xc420dbd980, 0xc461887b20, 0xc420dbd920) /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:439 +0xd1 created by github.com/influxdata/influxdb/services/httpd.(*Handler).serveQuery /go/src/github.com/influxdata/influxdb/services/httpd/handler.go:436 +0x1585

goroutine 10653 [IO wait]: internal/poll.runtime_pollWait(0x7f0c67bc7070, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc4200f6018, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc4200f6018, 0xc5ac355000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc4200f6000, 0xc5ac355000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc4200f6000, 0xc5ac355000, 0x1000, 0x1000, 0xc5bbc2f8c8, 0x68587b, 0xc4c240) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc5c65c2a60, 0xc5ac355000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc5c99220f0, 0xc5ac355000, 0x1000, 0x1000, 0x1506eee0beaf16b1, 0xc5bbc382c0, 0x13) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc496c0fda0) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc496c0fda0, 0xc515bced0a, 0x13, 0x0, 0x0, 0xfc, 0xfc) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc496c0fda0, 0x100, 0xf8, 0xca9c60, 0xc5bbc2fa98, 0x220000402db9, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc48fa7de60, 0xc5bbc2fb20, 0xc5bbc2fb20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc48fa7de60, 0xc57b9e7400, 0x0, 0xc5bbc2fb90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc496c0fda0, 0x0, 0xc57b9e7400, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc4212428c0, 0x10e24a0, 0xc43c22da00, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc4212428c0, 0x10e24a0, 0xc43c22da00) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10634 [IO wait, 1 minutes]: internal/poll.runtime_pollWait(0x7f0c67bc75b0, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc51a8cb418, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc51a8cb418, 0xc59909a000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc51a8cb400, 0xc59909a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc51a8cb400, 0xc59909a000, 0x1000, 0x1000, 0xc4216968c8, 0x68587b, 0xc4c240) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc491bd71f0, 0xc59909a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc53cdec630, 0xc59909a000, 0x1000, 0x1000, 0x18, 0xc421696a10, 0x406313) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc599b90f60) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc599b90f60, 0xc421696a0a, 0x411f36, 0x7f0c45b17990, 0x0, 0xf2, 0x32) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc599b90f60, 0x100, 0xf8, 0xca9c60, 0xc421696a01, 0x1220000402db9, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc5a6d88330, 0xc421696b20, 0xc421696b20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc5a6d88330, 0xc431754000, 0x0, 0xc421696b90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc599b90f60, 0x0, 0xc431754000, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc49b705f40, 0x10e24a0, 0xc5c5fd9600, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc49b705f40, 0x10e24a0, 0xc5c5fd9600) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10611 [IO wait, 1 minutes]: internal/poll.runtime_pollWait(0x7f0c67bc7430, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc5b3127c18, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc5b3127c18, 0xc42232b000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc5b3127c00, 0xc42232b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc5b3127c00, 0xc42232b000, 0x1000, 0x1000, 0xc42003f8c8, 0x68587b, 0xc4c240) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc42000ff28, 0xc42232b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc4214c9e30, 0xc42232b000, 0x1000, 0x1000, 0x18, 0xc42003fa10, 0x406313) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc492754060) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc492754060, 0xc422d9670a, 0xc42017c4d8, 0x0, 0x0, 0xc42003fab8, 0x4123c7) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc492754060, 0x100, 0xf8, 0xca9c60, 0x8, 0x2200202f0000, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc4214c9e60, 0xc42003fb20, 0xc42003fb20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc4214c9e60, 0xc494823d00, 0x0, 0xc42003fb90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc492754060, 0x0, 0xc494823d00, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc58d18c0a0, 0x10e24a0, 0xc5a6d77e40, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc58d18c0a0, 0x10e24a0, 0xc5a6d77e40) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 1757 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0xc4201d6b40, 0xc420dfa060) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1241 +0x129 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xc4201d6b40, 0xc420dfa060) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:317 +0x61 created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:317 +0x10b

goroutine 1758 [select]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0xc4201d6b40, 0xc420dfa660) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1281 +0x263 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0xc4201d6b40, 0xc420dfa660) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:267 +0x61 created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:267 +0x11f

goroutine 10643 [IO wait, 1 minutes]: internal/poll.runtime_pollWait(0x7f0c67bba060, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc539408798, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc539408798, 0xc424642000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc539408780, 0xc424642000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc539408780, 0xc424642000, 0x1000, 0x1000, 0xc4216978c8, 0x68587b, 0xc4c240) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc4941fa150, 0xc424642000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc48f26b080, 0xc424642000, 0x1000, 0x1000, 0x18, 0xc421697a10, 0x406313) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc51a0f7f20) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc51a0f7f20, 0x112040a, 0xc42017c4d8, 0x0, 0x0, 0xc421697ab8, 0x4123c7) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc51a0f7f20, 0x100, 0xf8, 0xca9c60, 0x8, 0x220020104a80, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc48f26b0b0, 0xc421697b20, 0xc421697b20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc48f26b0b0, 0xc5a4b9bd00, 0x0, 0xc421697b90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc51a0f7f20, 0x0, 0xc5a4b9bd00, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc58d18cfa0, 0x10e24a0, 0xc42fcb2740, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc58d18cfa0, 0x10e24a0, 0xc42fcb2740) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10565 [IO wait]: internal/poll.runtime_pollWait(0x7f0c67bc7a30, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc5a3046218, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc5a3046218, 0xc48c89a000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc5a3046200, 0xc48c89a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc5a3046200, 0xc48c89a000, 0x1000, 0x1000, 0xc5bbc2d8c8, 0x6858dc, 0xc5c3720c18) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc5c65c3a18, 0xc48c89a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc5c3720c00, 0xc48c89a000, 0x1000, 0x1000, 0x13, 0x4133f6, 0xc5bbc2d978) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc59eec8a20) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc59eec8a20, 0xc5c3720c0a, 0x7f0c459de840, 0x2, 0x41, 0xc5bbc2dab8, 0xc5bbc2da48) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc59eec8a20, 0x100, 0xf8, 0xca9c60, 0x4133f6, 0x2200bbc2da90, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc5c3720c30, 0xc5bbc2db20, 0xc5bbc2db20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc5c3720c30, 0xc53951b900, 0x0, 0xc5bbc2db90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc59eec8a20, 0x0, 0xc53951b900, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc6ba01c000, 0x10e24a0, 0xc584b76280, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc6ba01c000, 0x10e24a0, 0xc584b76280) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10644 [IO wait, 1 minutes]: internal/poll.runtime_pollWait(0x7f0c67bba120, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc539408818, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc539408818, 0xc51d5dd000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc539408800, 0xc51d5dd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc539408800, 0xc51d5dd000, 0x1000, 0x1000, 0xc4216b88c8, 0x68587b, 0xc4c240) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc4941fa158, 0xc51d5dd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc569688fc0, 0xc51d5dd000, 0x1000, 0x1000, 0x0, 0x10d14a0, 0xc420040060) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc4235474a0) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc4235474a0, 0xc42004000a, 0xc420040270, 0x0, 0x0, 0x30, 0xd9) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc4235474a0, 0x100, 0xf8, 0xca9c60, 0xc4216b8a98, 0x220000402db9, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc569688c30, 0xc4216b8b20, 0xc4216b8b20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc569688c30, 0xc42fcba200, 0x0, 0xc4216b8b90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc4235474a0, 0x0, 0xc42fcba200, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc58d18d040, 0x10e24a0, 0xc496894b40, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc58d18d040, 0x10e24a0, 0xc496894b40) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10613 [IO wait, 1 minutes]: internal/poll.runtime_pollWait(0x7f0c67bc7970, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc42fcac118, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc42fcac118, 0xc5c6ad1000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc42fcac100, 0xc5c6ad1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc42fcac100, 0xc5c6ad1000, 0x1000, 0x1000, 0xc5227e48c8, 0x68587b, 0xc4c240) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc491bd70c0, 0xc5c6ad1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc5ac3706f0, 0xc5c6ad1000, 0x1000, 0x1000, 0x0, 0x10d14a0, 0xc420040060) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc423547440) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc423547440, 0xc42004000a, 0xc420040270, 0x0, 0x0, 0xc5227e4ab8, 0x4123c7) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc423547440, 0x100, 0xf8, 0xca9c60, 0x8, 0x2200202f0000, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc5ac370720, 0xc5227e4b20, 0xc5227e4b20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc5ac370720, 0xc5a4b9ba00, 0x0, 0xc5227e4b90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc423547440, 0x0, 0xc5a4b9ba00, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc49b7050e0, 0x10e24a0, 0xc42fcb2140, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc49b7050e0, 0x10e24a0, 0xc42fcb2140) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10566 [IO wait]: internal/poll.runtime_pollWait(0x7f0c67bb9fa0, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc5a3046298, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc5a3046298, 0xc48c89c000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc5a3046280, 0xc48c89c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc5a3046280, 0xc48c89c000, 0x1000, 0x1000, 0xc5bbc2c8c8, 0x6858dc, 0xc5c3720b88) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc5c65c3a20, 0xc48c89c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc5c3720b70, 0xc48c89c000, 0x1000, 0x1000, 0x7f0c45ab0200, 0xc8, 0xc5bbc2c9c8) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc59eec89c0) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc59eec89c0, 0xc5c3720c0a, 0x7f0c459de840, 0x4, 0x42, 0xc5bbc2cab8, 0xc5bbc2ca48) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc59eec89c0, 0x100, 0xf8, 0xca9c60, 0x4133f6, 0x2200bbc2ca90, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc5c3720c60, 0xc5bbc2cb20, 0xc5bbc2cb20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc5c3720c60, 0xc53951ba00, 0x0, 0xc5bbc2cb90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc59eec89c0, 0x0, 0xc53951ba00, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc6ba01c0a0, 0x10e24a0, 0xc584b76240, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc6ba01c0a0, 0x10e24a0, 0xc584b76240) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10642 [IO wait, 1 minutes]: internal/poll.runtime_pollWait(0x7f0c67bc7bb0, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc48a23c098, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc48a23c098, 0xc491b49000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc48a23c080, 0xc491b49000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc48a23c080, 0xc491b49000, 0x1000, 0x1000, 0xc4216988c8, 0x68587b, 0xc4c240) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc422276160, 0xc491b49000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc5696883c0, 0xc491b49000, 0x1000, 0x1000, 0x0, 0x10d14a0, 0xc420040060) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc5803bd3e0) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc5803bd3e0, 0xc422ec290a, 0xc420040270, 0x0, 0x0, 0x30, 0xd9) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc5803bd3e0, 0x100, 0xf8, 0xca9c60, 0xc421698a98, 0x220000402db9, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc48fa7c150, 0xc421698b20, 0xc421698b20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc48fa7c150, 0xc42018db00, 0x0, 0xc421698b90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc5803bd3e0, 0x0, 0xc42018db00, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc48a240000, 0x10e24a0, 0xc494a97800, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc48a240000, 0x10e24a0, 0xc494a97800) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10703 [running]: goroutine running on other thread; stack unavailable created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createTagSetIterators /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1936 +0x53a

goroutine 10625 [IO wait]: internal/poll.runtime_pollWait(0x7f0c67bc7670, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc5b3126518, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc5b3126518, 0xc491b4b000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc5b3126500, 0xc491b4b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc5b3126500, 0xc491b4b000, 0x1000, 0x1000, 0xc5aa201b00, 0x68584a, 0xc59111c308) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc51c472000, 0xc491b4b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc59111c300, 0xc491b4b000, 0x1000, 0x1000, 0xc5aa201be8, 0x59aa0c, 0xc5b3126500) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc59eec8ea0) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).Peek(0xc59eec8ea0, 0x4, 0x0, 0x0, 0x0, 0x0, 0x7f0c67bc7600) /usr/local/go/src/bufio/bufio.go:129 +0x3a net/http.(*conn).readRequest(0xc421c80140, 0x10e24a0, 0xc421946200, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:930 +0xc77 net/http.(*conn).serve(0xc421c80140, 0x10e24a0, 0xc421946200) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10694 [semacquire]: sync.runtime_Semacquire(0xc584abd2dc) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(*WaitGroup).Wait(0xc584abd2d0) /usr/local/go/src/sync/waitgroup.go:131 +0x72 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createTagSetIterators(0xc42044a7e0, 0x10e24e0, 0xc4200140c0, 0x0, 0xc584abd0d0, 0xe, 0xc48f8d6eb0, 0x0, 0x0, 0xc5159b9180, …) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1941 +0x568 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createVarRefIterator.func1(0xc42000f588, 0x1, 0x1, 0xc42044a7e0, 0x10e24e0, 0xc4200140c0, 0x0, 0xc584abd0d0, 0xe, 0xc49aee04b0, …) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1853 +0x12b github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).createVarRefIterator(0xc42044a7e0, 0x10e24e0, 0xc4200140c0, 0xc584abd0d0, 0xe, 0x0, 0x0, 0xc5159b9180, 0x9, 0x9, …) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1894 +0x32a github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).CreateIterator(0xc42044a7e0, 0x10e24e0, 0xc4200140c0, 0xc584abd0d0, 0xe, 0x0, 0x0, 0xc5159b9180, 0x9, 0x9, …) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1748 +0x5b8 github.com/influxdata/influxdb/tsdb.(*Shard).CreateIterator(0xc42045ab60, 0x10e24e0, 0xc4200140c0, 0xc48f8d6dc0, 0x0, 0x0, 0xc5159b9180, 0x9, 0x9, 0x0, …) /go/src/github.com/influxdata/influxdb/tsdb/shard.go:835 +0x13c github.com/influxdata/influxdb/tsdb.Shards.CreateIterator(0xc5c5ce4670, 0x2, 0x2, 0x10e24e0, 0xc4200140c0, 0xc48f8d6dc0, 0x0, 0x0, 0xc5159b9180, 0x9, …) /go/src/github.com/influxdata/influxdb/tsdb/shard.go:1243 +0x130 github.com/influxdata/influxdb/tsdb.(*Shards).CreateIterator(0xc5846cf920, 0x10e24e0, 0xc4200140c0, 0xc48f8d6dc0, 0x0, 0x0, 0xc5159b9180, 0x9, 0x9, 0x0, …) <autogenerated>:1 +0xb5 github.com/influxdata/influxdb/coordinator.(*LocalShardMapping).CreateIterator(0xc582ca28c0, 0x10e24e0, 0xc4200140c0, 0xc48f8d6dc0, 0x0, 0x0, 0xc5159b9180, 0x9, 0x9, 0x0, …) /go/src/github.com/influxdata/influxdb/coordinator/shard_mapper.go:208 +0x46e github.com/influxdata/influxdb/query.buildAuxIterators.func1(0xc5c5ce4680, 0x1, 0x1, 0x7f0c64072618, 0xc582ca28c0, 0x10e24e0, 0xc4200140c0, 0xc49aee1280, 0xc49aee1208, 0x0, …) /go/src/github.com/influxdata/influxdb/query/select.go:199 +0x134 github.com/influxdata/influxdb/query.buildAuxIterators(0x10e24e0, 0xc4200140c0, 0xc48f8d6e10, 0x9, 0x9, 0x7f0c64072618, 0xc582ca28c0, 0xc5c5ce4680, 0x1, 0x1, …) /go/src/github.com/influxdata/influxdb/query/select.go:218 +0xf8 github.com/influxdata/influxdb/query.buildIterators(0x10e24e0, 0xc4200140c0, 0xc43ef12400, 0x7f0c64072618, 0xc582ca28c0, 0x0, 0x0, 0xc5159b9180, 0x9, 0x9, …) /go/src/github.com/influxdata/influxdb/query/select.go:138 +0x10ac github.com/influxdata/influxdb/query.(*preparedStatement).Select(0xc596e8a640, 0x10e24e0, 0xc4200140c0, 0x10e4060, 0x1149230, 0x0, 0xc420dbda40, 0x0, 0x0, 0x10de4a0, …) /go/src/github.com/influxdata/influxdb/query/select.go:103 +0xd0 github.com/influxdata/influxdb/query.Select(0x10e24e0, 0xc4200140c0, 0xc43ef12100, 0x10d17e0, 0xc42015faa0, 0x10e4060, 0x1149230, 0x0, 0xc420dbda40, 0x0, …) /go/src/github.com/influxdata/influxdb/query/select.go:89 +0x148 github.com/influxdata/influxdb/coordinator.(*StatementExecutor).createIterators(0xc420218a80, 0x10e24e0, 0xc4200140c0, 0xc43ef12100, 0xc49aee1cb0, 0x0, 0x0, 0x800c5cd9400, 0x1, 0xc420010108, …) /go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:659 +0x106 github.com/influxdata/influxdb/coordinator.(*StatementExecutor).executeSelectStatement(0xc420218a80, 0x10e24e0, 0xc4200140c0, 0xc43ef12100, 0xc49aee1cb0, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:555 +0x8f github.com/influxdata/influxdb/coordinator.(*StatementExecutor).ExecuteStatement(0xc420218a80, 0x10e38e0, 0xc43ef12100, 0x0, 0x16, 0xc4235f36c0, 0xc420dbd9e0, 0x10e7780, 0xc420224600, 0xc420dbda40, …) /go/src/github.com/influxdata/influxdb/coordinator/statement_executor.go:62 +0x29a6 github.com/influxdata/influxdb/query.(*QueryExecutor).executeQuery(0xc420155c20, 0xc5846cf800, 0xc5c5cde29c, 0xe, 0x10e4060, 0x1149230, 0x2710, 0x0, 0x0, 0x0, …) /go/src/github.com/influxdata/influxdb/query/query_executor.go:381 +0xb27 created by github.com/influxdata/influxdb/query.(*QueryExecutor).ExecuteQuery /go/src/github.com/influxdata/influxdb/query/query_executor.go:276 +0xc3

goroutine 10675 [IO wait]: internal/poll.runtime_pollWait(0x7f0c67bba1e0, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc56a6ea918, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc56a6ea918, 0xc490cab000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc56a6ea900, 0xc490cab000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc56a6ea900, 0xc490cab000, 0x1000, 0x1000, 0xc5bbc308c8, 0x68587b, 0xc4c240) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc44fbcdba8, 0xc490cab000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc5c7550c00, 0xc490cab000, 0x1000, 0x1000, 0x8c80, 0xc5b633d260, 0x10d68e0) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc5726f1260) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc5726f1260, 0xc5bbc30a0a, 0x411f36, 0x7f0c67bf1b80, 0x2, 0xfc, 0xfc) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc5726f1260, 0x100, 0xf8, 0xca9c60, 0xc5bbc30a01, 0x1220000402db9, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc59111c330, 0xc5bbc30b20, 0xc5bbc30b20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc59111c330, 0xc5a3536200, 0x0, 0xc5bbc30b90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc5726f1260, 0x0, 0xc5a3536200, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc4210c1180, 0x10e24a0, 0xc51a876280, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc4210c1180, 0x10e24a0, 0xc51a876280) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10695 [select]: github.com/influxdata/influxdb/query.(*TaskManager).waitForQuery(0xc42017de00, 0x16, 0xc420dbda40, 0xc420dbd920, 0xc420dbdaa0) /go/src/github.com/influxdata/influxdb/query/task_manager.go:259 +0x166 created by github.com/influxdata/influxdb/query.(*TaskManager).AttachQuery /go/src/github.com/influxdata/influxdb/query/task_manager.go:176 +0x335

goroutine 10618 [IO wait, 1 minutes]: internal/poll.runtime_pollWait(0x7f0c67bc7130, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc57b9dc918, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc57b9dc918, 0xc580212000, 0x1000, 0x1000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc57b9dc900, 0xc580212000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc57b9dc900, 0xc580212000, 0x1000, 0x1000, 0xc4216938c8, 0x68587b, 0xc4c240) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc4222765d0, 0xc580212000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).Read(0xc435488b70, 0xc580212000, 0x1000, 0x1000, 0x0, 0x10d14a0, 0xc420040060) /usr/local/go/src/net/http/server.go:753 +0x105 bufio.(*Reader).fill(0xc45ac64300) /usr/local/go/src/bufio/bufio.go:97 +0x11a bufio.(*Reader).ReadSlice(0xc45ac64300, 0xc42004000a, 0xc420040270, 0x0, 0x0, 0xc421693ab8, 0x4123c7) /usr/local/go/src/bufio/bufio.go:338 +0x2c bufio.(*Reader).ReadLine(0xc45ac64300, 0x100, 0xf8, 0xca9c60, 0x8, 0x22002002e700, 0xf8) /usr/local/go/src/bufio/bufio.go:367 +0x34 net/textproto.(*Reader).readLineSlice(0xc435488ba0, 0xc421693b20, 0xc421693b20, 0x412b28, 0x100, 0xca9c60) /usr/local/go/src/net/textproto/reader.go:55 +0x70 net/textproto.(*Reader).ReadLine(0xc435488ba0, 0xc43174e900, 0x0, 0xc421693b90, 0x494e22) /usr/local/go/src/net/textproto/reader.go:36 +0x2b net/http.readRequest(0xc45ac64300, 0x0, 0xc43174e900, 0x0, 0x0) /usr/local/go/src/net/http/request.go:925 +0x99 net/http.(*conn).readRequest(0xc58d18dea0, 0x10e24a0, 0xc496e5a6c0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:933 +0x17c net/http.(*conn).serve(0xc58d18dea0, 0x10e24a0, 0xc496e5a6c0) /usr/local/go/src/net/http/server.go:1739 +0x50e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2720 +0x288

goroutine 10692 [IO wait]: internal/poll.runtime_pollWait(0x7f0c67bc7370, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc421c8e218, 0x72, 0xffffffffffffff00, 0x10d72a0, 0x10cf6e8) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc421c8e218, 0xc420228f00, 0x1, 0x1) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc421c8e200, 0xc420228fa1, 0x1, 0x1, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a net.(*netFD).Read(0xc421c8e200, 0xc420228fa1, 0x1, 0x1, 0xc45f1de640, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc4204642a0, 0xc420228fa1, 0x1, 0x1, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d net/http.(*connReader).backgroundRead(0xc420228f90) /usr/local/go/src/net/http/server.go:660 +0x62 created by net/http.(*connReader).startBackgroundRead /usr/local/go/src/net/http/server.go:656 +0xd8`