thanos: compact: 0.22.0 never starts downsampling work
I’m running Thanos 0.22.0 binary from GH releases
$ thanos --version
thanos, version 0.22.0 (branch: HEAD, revision: 3656e290c8eb034e2e09e0afc79f087be88de481)
build user: root@61bcfbac7168
build date: 20210721-15:22:37
go version: go1.16.6
platform: linux/amd64
Object Storage Provider: swift + s3api compat
What happened:
Thanos compact stops making progress after "start first pass of downsampling" message, e.g. I don’t see any download block log from processDownsampling. I’m running compact like this (i.e. compact.concurrency=1 and default downsample concurrency):
/usr/bin/thanos compact --http-address 0.0.0.0:12902 --data-dir /srv/thanos-compact --objstore.config-file /etc/thanos-compact/objstore.yaml --retention.resolution-raw 270w --retention.resolution-5m 270w --retention.resolution-1h 270w --compact.concurrency 1 --wait
I’m seeing this in the logs and then nothing ever again from compact.go, though I’m quite sure there is downsampling work to do:
Aug 07 09:33:36 thanos-fe2001 thanos-compact[2624124]: level=info ts=2021-08-07T09:33:36.462956677Z caller=compact.go:976 msg="start of compactions"
Aug 07 09:33:36 thanos-fe2001 thanos-compact[2624124]: level=info ts=2021-08-07T09:33:36.467690366Z caller=compact.go:1008 msg="compaction iterations done"
Aug 07 09:33:36 thanos-fe2001 thanos-compact[2624124]: level=info ts=2021-08-07T09:33:36.467817352Z caller=compact.go:420 msg="start first pass of downsampling"
See below for full goroutine dump, I suspect this is due to the recent downsample concurrency change, cc @yeya24.
goroutine 1076625 [select, 16 minutes]:
main.downsampleBucket.func4(0x0, 0x0)
/app/cmd/thanos/downsample.go:297 +0x25e
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0021d4840, 0xc002b04000)
/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x66
goroutine 260 [semacquire, 16 minutes]:
sync.runtime_Semacquire(0xc0021d4850)
/usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc0021d4848)
/usr/local/go/src/sync/waitgroup.go:130 +0x65
golang.org/x/sync/errgroup.(*Group).Wait(0xc0021d4840, 0xc002b04000, 0x4)
/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:40 +0x31
main.downsampleBucket(0x201fee8, 0xc00034fc00, 0x1fedfa0, 0xc000632210, 0xc00055ce10, 0x203c118, 0xc00055dca0, 0xc0049beb40, 0xc000318c80, 0x1e, ...)
/app/cmd/thanos/downsample.go:306 +0xcc5
main.runCompact.func7(0xc000bced80, 0x458d29)
/app/cmd/thanos/compact.go:430 +0x8b1
main.runCompact.func8.1(0x45d964b800, 0xc0003c6000)
/app/cmd/thanos/compact.go:467 +0x5c
github.com/thanos-io/thanos/pkg/runutil.Repeat(0x45d964b800, 0xc000bc6000, 0xc0007a3f08, 0x0, 0x0)
/app/pkg/runutil/runutil.go:75 +0x88
main.runCompact.func8(0x0, 0x0)
/app/cmd/thanos/compact.go:466 +0x1ff
github.com/oklog/run.(*Group).Run.func1(0xc000361ec0, 0xc000363630, 0xc000451490)
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xbb
What you expected to happen:
Compact making progress
How to reproduce it (as minimally and precisely as possible):
Wait for compact to start downsampling pass
Full logs to relevant components:
Logs
goroutine 1174721 [running]:
runtime/pprof.writeGoroutineStacks(0x1ff1dc0, 0xc000c400e0, 0x30, 0x1b5a800)
/usr/local/go/src/runtime/pprof/pprof.go:693 +0x9f
runtime/pprof.writeGoroutine(0x1ff1dc0, 0xc000c400e0, 0x2, 0xc0004f7000, 0x0)
/usr/local/go/src/runtime/pprof/pprof.go:682 +0x45
runtime/pprof.(*Profile).WriteTo(0x2d9b640, 0x1ff1dc0, 0xc000c400e0, 0x2, 0xc000c400e0, 0x7b9898)
/usr/local/go/src/runtime/pprof/pprof.go:331 +0x3f2
net/http/pprof.handler.ServeHTTP(0xc002790011, 0x9, 0x2019060, 0xc000c400e0, 0xc0008a8200)
/usr/local/go/src/net/http/pprof/pprof.go:253 +0x385
net/http/pprof.Index(0x2019060, 0xc000c400e0, 0xc0008a8200)
/usr/local/go/src/net/http/pprof/pprof.go:371 +0x8d7
net/http.HandlerFunc.ServeHTTP(0x1da51f0, 0x2019060, 0xc000c400e0, 0xc0008a8200)
/usr/local/go/src/net/http/server.go:2049 +0x44
net/http.(*ServeMux).ServeHTTP(0xc00034f340, 0x2019060, 0xc000c400e0, 0xc0008a8200)
/usr/local/go/src/net/http/server.go:2428 +0x1ad
net/http.serverHandler.ServeHTTP(0xc000330000, 0x2019060, 0xc000c400e0, 0xc0008a8200)
/usr/local/go/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc0003c8140, 0x201ff90, 0xc00426e0c0)
/usr/local/go/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2993 +0x39b
goroutine 1 [chan receive, 168 minutes]:
github.com/oklog/run.(*Group).Run(0xc0003010b0, 0xc000393ed0, 0xc000434c00)
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:43 +0xed
main.main()
/app/cmd/thanos/main.go:154 +0xd68
goroutine 49 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0003a6080)
/go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xcd
created by go.opencensus.io/stats/view.init.0
/go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x68
goroutine 258 [chan receive, 168 minutes]:
main.main.func2(0xc000591fb8, 0x93361b)
/app/cmd/thanos/main.go:114 +0x50
github.com/oklog/run.(*Group).Run.func1(0xc000361ec0, 0xc0003010c8, 0xc000632630)
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xbb
goroutine 259 [IO wait]:
internal/poll.runtime_pollWait(0x7fb90ccd8620, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000376118, 0x72, 0x0, 0x0, 0x1d00c25)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000376100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:401 +0x212
net.(*netFD).accept(0xc000376100, 0xc04c2fae0b16dc85, 0x0, 0x0)
/usr/local/go/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc00043a018, 0x610e57af, 0xc0002a8c18, 0x497686)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc00043a018, 0xc0002a8c68, 0x18, 0xc000652900, 0x7bbefb)
/usr/local/go/src/net/tcpsock.go:261 +0x65
net/http.(*Server).Serve(0xc000330000, 0x2018e20, 0xc00043a018, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2961 +0x285
github.com/prometheus/exporter-toolkit/web.Serve(0x2018e20, 0xc00043a018, 0xc000330000, 0x0, 0x0, 0x1fedfa0, 0xc000632c90, 0x0, 0xc000bc2b40)
/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.5.1/web/tls_config.go:192 +0x1b0
github.com/prometheus/exporter-toolkit/web.ListenAndServe(0xc000330000, 0x0, 0x0, 0x1fedfa0, 0xc000632c90, 0x0, 0x0)
/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.5.1/web/tls_config.go:184 +0x105
github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe(0xc000b5ad90, 0xc000434b98, 0xc0002b7cd0)
/app/pkg/server/http/http.go:68 +0x216
main.runCompact.func1(0x1da5530, 0xc000594fc0)
/app/cmd/thanos/compact.go:189 +0x46
github.com/oklog/run.(*Group).Run.func1(0xc000361ec0, 0xc000303080, 0xc0003030a0)
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xbb
goroutine 260 [semacquire, 16 minutes]:
sync.runtime_Semacquire(0xc0021d4850)
/usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc0021d4848)
/usr/local/go/src/sync/waitgroup.go:130 +0x65
golang.org/x/sync/errgroup.(*Group).Wait(0xc0021d4840, 0xc002b04000, 0x4)
/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:40 +0x31
main.downsampleBucket(0x201fee8, 0xc00034fc00, 0x1fedfa0, 0xc000632210, 0xc00055ce10, 0x203c118, 0xc00055dca0, 0xc0049beb40, 0xc000318c80, 0x1e, ...)
/app/cmd/thanos/downsample.go:306 +0xcc5
main.runCompact.func7(0xc000bced80, 0x458d29)
/app/cmd/thanos/compact.go:430 +0x8b1
main.runCompact.func8.1(0x45d964b800, 0xc0003c6000)
/app/cmd/thanos/compact.go:467 +0x5c
github.com/thanos-io/thanos/pkg/runutil.Repeat(0x45d964b800, 0xc000bc6000, 0xc0007a3f08, 0x0, 0x0)
/app/pkg/runutil/runutil.go:75 +0x88
main.runCompact.func8(0x0, 0x0)
/app/cmd/thanos/compact.go:466 +0x1ff
github.com/oklog/run.(*Group).Run.func1(0xc000361ec0, 0xc000363630, 0xc000451490)
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xbb
goroutine 261 [select, 2 minutes]:
github.com/thanos-io/thanos/pkg/runutil.Repeat(0x45d964b800, 0xc000bc6000, 0xc000119920, 0x0, 0x0)
/app/pkg/runutil/runutil.go:78 +0xee
main.runCompact.func12(0x0, 0x0)
/app/cmd/thanos/compact.go:530 +0x65
github.com/oklog/run.(*Group).Run.func1(0xc000361ec0, 0xc0002b7dd0, 0xc000393e20)
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xbb
goroutine 262 [select]:
github.com/thanos-io/thanos/pkg/runutil.Repeat(0xdf8475800, 0xc000bc6000, 0xc00062df50, 0x0, 0x0)
/app/pkg/runutil/runutil.go:78 +0xee
main.runCompact.func14(0x0, 0x0)
/app/cmd/thanos/compact.go:542 +0x165
github.com/oklog/run.(*Group).Run.func1(0xc000361ec0, 0xc000b5d980, 0xc000393e30)
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xbb
goroutine 263 [select, 168 minutes]:
main.interrupt(0x1fedfa0, 0xc000632210, 0xc000434ba0, 0x0, 0x0)
/app/cmd/thanos/main.go:165 +0x135
main.main.func4(0x0, 0x0)
/app/cmd/thanos/main.go:138 +0x3c
github.com/oklog/run.(*Group).Run.func1(0xc000361ec0, 0xc000303d60, 0xc000393ec0)
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xbb
goroutine 264 [select, 168 minutes]:
main.reload(0x1fedfa0, 0xc000632210, 0xc000434c00, 0xc0000a6720, 0x0, 0x0)
/app/cmd/thanos/main.go:178 +0x11b
main.main.func6(0x0, 0x0)
/app/cmd/thanos/main.go:148 +0x45
github.com/oklog/run.(*Group).Run.func1(0xc000361ec0, 0xc0002b7f20, 0xc000393ed0)
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x27
created by github.com/oklog/run.(*Group).Run
/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0xbb
goroutine 266 [syscall, 168 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:168 +0xa5
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:151 +0x45
goroutine 1114999 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc000f97560)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1087743 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc001933320)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1030962 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc004fbec60)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1150981 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0006ac000)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1119352 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc00048d440)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1095136 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc002d6a480)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1151012 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90ccd7fc8, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc004f4a898, 0x72, 0x900, 0x952, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc004f4a880, 0xc00424e000, 0x952, 0x952, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc004f4a880, 0xc00424e000, 0x952, 0x952, 0x94d, 0xc00424e000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc003614238, 0xc00424e000, 0x952, 0x952, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312d80, 0xc00424e000, 0x952, 0x952, 0xc002a609f8, 0xc000100000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000b385f8, 0x1fed400, 0xc003312d80, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000b38380, 0x1ff1aa0, 0xc003614238, 0x5, 0xc003614238, 0xc002a60a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000b38380, 0x0, 0x0, 0xc002a60e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000b38380, 0xc00427e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0016bb9e0, 0xc00427e000, 0x1000, 0x1000, 0xc002a60dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0021c0780)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0021c0780, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0016bb9e0)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1149512 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0043b0120)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1030961 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90c7a0a08, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0052de498, 0x72, 0xcd00, 0xcd94, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0052de480, 0xc006654000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0052de480, 0xc006654000, 0xcd94, 0xcd94, 0xcd8f, 0xc006654000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000930000, 0xc006654000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc00094dc50, 0xc006654000, 0xcd94, 0xcd94, 0xc0007dd9f8, 0xc000650400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc00317c278, 0x1fed400, 0xc00094dc50, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc00317c000, 0x1ff1aa0, 0xc000930000, 0x5, 0xc000930000, 0xc0007dda40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc00317c000, 0x0, 0x0, 0xc0007dde88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc00317c000, 0xc005068000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc004fbec60, 0xc005068000, 0x1000, 0x1000, 0xc0007dddc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc004b881e0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc004b881e0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc004fbec60)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1119350 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc000de3560)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1119278 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0006e9320)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1114998 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90ccd7888, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc004cb8a18, 0x72, 0xcd00, 0xcd92, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc004cb8a00, 0xc00291c000, 0xcd92, 0xcd92, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc004cb8a00, 0xc00291c000, 0xcd92, 0xcd92, 0xcd8d, 0xc00291c000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00033a3b0, 0xc00291c000, 0xcd92, 0xcd92, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0034d4408, 0xc00291c000, 0xcd92, 0xcd92, 0xc0017899f8, 0xc000100000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc00317c978, 0x1fed400, 0xc0034d4408, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc00317c700, 0x1ff1aa0, 0xc00033a3b0, 0x5, 0xc00033a3b0, 0xc001789a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc00317c700, 0x0, 0x0, 0xc001789e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc00317c700, 0xc004a00000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc000f97560, 0xc004a00000, 0x1000, 0x1000, 0xc001789dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc005353c20)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc005353c20, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc000f97560)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1151060 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cd41f68, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0007e6998, 0x72, 0x900, 0x952, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0007e6980, 0xc00468aa80, 0x952, 0x952, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0007e6980, 0xc00468aa80, 0x952, 0x952, 0x94d, 0xc00468aa80, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0053a1388, 0xc00468aa80, 0x952, 0x952, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312c90, 0xc00468aa80, 0x952, 0x952, 0xc0022aa9f8, 0xc000100000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000140978, 0x1fed400, 0xc003312c90, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000140700, 0x1ff1aa0, 0xc0053a1388, 0x5, 0xc0053a1388, 0xc0022aaa40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000140700, 0x0, 0x0, 0xc0022aae88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000140700, 0xc0046b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0043b0360, 0xc0046b8000, 0x1000, 0x1000, 0xc0022aadc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0057a3c80)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0057a3c80, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0043b0360)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1150816 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0015e2120)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1150952 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0043b0000)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1119409 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0043b10e0)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1151028 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0016bb7a0)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1150947 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cd41910, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0003a6b18, 0x72, 0xcd00, 0xcd93, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0003a6b00, 0xc002748000, 0xcd93, 0xcd93, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0003a6b00, 0xc002748000, 0xcd93, 0xcd93, 0xcd8e, 0xc002748000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0053a13a8, 0xc002748000, 0xcd93, 0xcd93, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0034d4348, 0xc002748000, 0xcd93, 0xcd93, 0xc001df69f8, 0xc000650400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000bb8278, 0x1fed400, 0xc0034d4348, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000bb8000, 0x1ff1aa0, 0xc0053a13a8, 0x5, 0xc0053a13a8, 0xc001df6a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000bb8000, 0x0, 0x0, 0xc001df6e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000bb8000, 0xc00336a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0016bb560, 0xc00336a000, 0x1000, 0x1000, 0xc001df6dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc005d31ec0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc005d31ec0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0016bb560)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1151029 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90ccd8538, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0003a6f98, 0x72, 0x4b00, 0x4b66, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0003a6f80, 0xc002d24000, 0x4b66, 0x4b66, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0003a6f80, 0xc002d24000, 0x4b66, 0x4b66, 0x4b61, 0xc002d24000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc004e594b0, 0xc002d24000, 0x4b66, 0x4b66, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312d50, 0xc002d24000, 0x4b66, 0x4b66, 0xc001d5b9f8, 0xc000708800, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000b7c5f8, 0x1fed400, 0xc003312d50, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000b7c380, 0x1ff1aa0, 0xc004e594b0, 0x5, 0xc004e594b0, 0xc001d5ba40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000b7c380, 0x0, 0x0, 0xc001d5be88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000b7c380, 0xc004a94000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0053ae480, 0xc004a94000, 0x1000, 0x1000, 0xc001d5bdc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc00066c4e0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc00066c4e0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0053ae480)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1150840 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0020425a0)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1150951 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90c792528, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc004cb8398, 0x72, 0x900, 0x950, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc004cb8380, 0xc00337e000, 0x950, 0x950, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc004cb8380, 0xc00337e000, 0x950, 0x950, 0x94b, 0xc00337e000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0005d86a8, 0xc00337e000, 0x950, 0x950, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0034d43d8, 0xc00337e000, 0x950, 0x950, 0xc0003c19f8, 0xc000708800, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc004af0cf8, 0x1fed400, 0xc0034d43d8, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc004af0a80, 0x1ff1aa0, 0xc0005d86a8, 0x5, 0xc0005d86a8, 0xc0003c1a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc004af0a80, 0x0, 0x0, 0xc0003c1e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc004af0a80, 0xc004a0a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0043b0000, 0xc004a0a000, 0x1000, 0x1000, 0xc0003c1dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0048cacc0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0048cacc0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0043b0000)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1150963 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90ccd7d10, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc004f4a498, 0x72, 0x900, 0x951, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc004f4a480, 0xc004547500, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc004f4a480, 0xc004547500, 0x951, 0x951, 0x94c, 0xc004547500, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc003614248, 0xc004547500, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312e10, 0xc004547500, 0x951, 0x951, 0xc006db99f8, 0xc000650400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000b39778, 0x1fed400, 0xc003312e10, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000b39500, 0x1ff1aa0, 0xc003614248, 0x5, 0xc003614248, 0xc006db9a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000b39500, 0x0, 0x0, 0xc006db9e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000b39500, 0xc001aea000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0043b0480, 0xc001aea000, 0x1000, 0x1000, 0xc006db9dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc006511140)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc006511140, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0043b0480)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 952094 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cd41d98, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000376918, 0x72, 0xc600, 0xc618, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000376900, 0xc001036000, 0xc618, 0xc618, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000376900, 0xc001036000, 0xc618, 0xc618, 0xc613, 0xc001036000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00033a058, 0xc001036000, 0xc618, 0xc618, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0034d4288, 0xc001036000, 0xc618, 0xc618, 0xc00178a9f8, 0xc0004f6400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc0001413f8, 0x1fed400, 0xc0034d4288, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000141180, 0x1ff1aa0, 0xc00033a058, 0x5, 0xc00033a058, 0xc00178aa40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000141180, 0x0, 0x0, 0xc00178ae88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000141180, 0xc0018a2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0006ac7e0, 0xc0018a2000, 0x1000, 0x1000, 0xc00178adc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0038de900)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0038de900, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0006ac7e0)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 18359 [IO wait]:
internal/poll.runtime_pollWait(0x7fb90c7a1230, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000792318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000792300, 0xc00463b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000792300, 0xc00463b000, 0x1000, 0x1000, 0x7b5836, 0xc00055e000, 0xf0)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00011e300, 0xc00463b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
net/http.(*connReader).Read(0xc00462efc0, 0xc00463b000, 0x1000, 0x1000, 0x8, 0xc000ce3780, 0x7aed09)
/usr/local/go/src/net/http/server.go:780 +0x1b9
bufio.(*Reader).fill(0xc004644060)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).ReadSlice(0xc004644060, 0xa, 0x1e, 0x100, 0x8, 0xc0003581a0, 0x7b5836)
/usr/local/go/src/bufio/bufio.go:360 +0x3d
bufio.(*Reader).ReadLine(0xc004644060, 0x203000, 0x203000, 0xc000ce38d0, 0x40db9b, 0xc001c19290, 0x30)
/usr/local/go/src/bufio/bufio.go:389 +0x34
net/textproto.(*Reader).readLineSlice(0xc001c19290, 0xc000b1b900, 0xc000ce39e8, 0x4e9393, 0xc000792300, 0x43b03c)
/usr/local/go/src/net/textproto/reader.go:57 +0xd6
net/textproto.(*Reader).ReadLine(...)
/usr/local/go/src/net/textproto/reader.go:38
net/http.readRequest(0xc004644060, 0x0, 0xc000b1b900, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:1027 +0xaa
net/http.(*conn).readRequest(0xc004610960, 0x201fee8, 0xc004634fc0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:966 +0x19d
net/http.(*conn).serve(0xc004610960, 0x201ff90, 0xc004634fc0)
/usr/local/go/src/net/http/server.go:1858 +0x705
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2993 +0x39b
goroutine 1119303 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0006e9200)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1150814 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0043b0240)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1150839 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90c792a98, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc00320c298, 0x72, 0x900, 0x951, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00320c280, 0xc0004b4000, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc00320c280, 0xc0004b4000, 0x951, 0x951, 0x94c, 0xc0004b4000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000fb62c8, 0xc0004b4000, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0044c9680, 0xc0004b4000, 0x951, 0x951, 0xc002a5f9f8, 0xc00007b000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000b46278, 0x1fed400, 0xc0044c9680, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000b46000, 0x1ff1aa0, 0xc000fb62c8, 0x5, 0xc000fb62c8, 0xc002a5fa40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000b46000, 0x0, 0x0, 0xc002a5fe88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000b46000, 0xc0045d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0020425a0, 0xc0045d0000, 0x1000, 0x1000, 0xc002a5fdc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc002a4e6c0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc002a4e6c0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0020425a0)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 16886 [IO wait]:
internal/poll.runtime_pollWait(0x7fb90c7a1318, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc002812118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc002812100, 0xc0034fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc002812100, 0xc0034fe000, 0x1000, 0x1000, 0x7b5836, 0xc0003300e0, 0xf0)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000b16a88, 0xc0034fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
net/http.(*connReader).Read(0xc0069ff4d0, 0xc0034fe000, 0x1000, 0x1000, 0x8, 0xc00059f780, 0x7aed09)
/usr/local/go/src/net/http/server.go:780 +0x1b9
bufio.(*Reader).fill(0xc006cb5860)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).ReadSlice(0xc006cb5860, 0xa, 0x50bc, 0x41909b, 0x7fb90c79a910, 0x7fb90c9521a0, 0x0)
/usr/local/go/src/bufio/bufio.go:360 +0x3d
bufio.(*Reader).ReadLine(0xc006cb5860, 0x203000, 0x203000, 0x7fb90c79a910, 0xa, 0xc004689e30, 0x30)
/usr/local/go/src/bufio/bufio.go:389 +0x34
net/textproto.(*Reader).readLineSlice(0xc004689e30, 0xc0015b4a00, 0xc00059f9e8, 0x4e9393, 0xc002812100, 0x43b03c)
/usr/local/go/src/net/textproto/reader.go:57 +0xd6
net/textproto.(*Reader).ReadLine(...)
/usr/local/go/src/net/textproto/reader.go:38
net/http.readRequest(0xc006cb5860, 0x0, 0xc0015b4a00, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:1027 +0xaa
net/http.(*conn).readRequest(0xc0034a8a00, 0x201fee8, 0xc0034ebe40, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:966 +0x19d
net/http.(*conn).serve(0xc0034a8a00, 0x201ff90, 0xc0034ebe40)
/usr/local/go/src/net/http/server.go:1858 +0x705
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2993 +0x39b
goroutine 1151030 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0053ae480)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1150949 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cca7528, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000130898, 0x72, 0x900, 0x952, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000130880, 0xc00541f500, 0x952, 0x952, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000130880, 0xc00541f500, 0x952, 0x952, 0x94d, 0xc00541f500, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc003614280, 0xc00541f500, 0x952, 0x952, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc00094dbf0, 0xc00541f500, 0x952, 0x952, 0xc0037a59f8, 0xc000650400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000b39af8, 0x1fed400, 0xc00094dbf0, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000b39880, 0x1ff1aa0, 0xc003614280, 0x5, 0xc003614280, 0xc0037a5a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000b39880, 0x0, 0x0, 0xc0037a5e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000b39880, 0xc003372000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0021bcd80, 0xc003372000, 0x1000, 0x1000, 0xc0037a5dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0065755c0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0065755c0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0021bcd80)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1150813 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cd41740, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc004cb8298, 0x72, 0xcd00, 0xcd93, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc004cb8280, 0xc002828000, 0xcd93, 0xcd93, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc004cb8280, 0xc002828000, 0xcd93, 0xcd93, 0xcd8e, 0xc002828000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000c6a5f0, 0xc002828000, 0xcd93, 0xcd93, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc00094db18, 0xc002828000, 0xcd93, 0xcd93, 0xc0003449f8, 0xc000100000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc0009ac5f8, 0x1fed400, 0xc00094db18, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc0009ac380, 0x1ff1aa0, 0xc000c6a5f0, 0x5, 0xc000c6a5f0, 0xc000344a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0009ac380, 0x0, 0x0, 0xc000344e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc0009ac380, 0xc0042d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0043b0240, 0xc0042d8000, 0x1000, 0x1000, 0xc000344dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0050bcf00)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0050bcf00, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0043b0240)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1076598 [select, 16 minutes]:
github.com/minio/minio-go/v7.Client.listObjectsV2.func1(0xc00387e000, 0x201fee8, 0xc00034fc00, 0xc00026c608, 0x6, 0xc001ad8080, 0x1b, 0x4b0001, 0x1fc1014, 0x1, ...)
/go/pkg/mod/github.com/bwplotka/minio-go/v7@v7.0.11-0.20210324165441-f9927e5255a6/api-list.go:123 +0x3db
created by github.com/minio/minio-go/v7.Client.listObjectsV2
/go/pkg/mod/github.com/bwplotka/minio-go/v7@v7.0.11-0.20210324165441-f9927e5255a6/api-list.go:93 +0x2ba
goroutine 1119305 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc00172b320)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1150843 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90ccd8368, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc005666218, 0x72, 0x900, 0x951, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005666200, 0xc0004b5500, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc005666200, 0xc0004b5500, 0x951, 0x951, 0x20300000000000, 0x7fb90d071fff, 0x400)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000b16080, 0xc0004b5500, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0034d42b8, 0xc0004b5500, 0x951, 0x951, 0xc000d9a9f8, 0xc0004f6400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc0008c13f8, 0x1fed400, 0xc0034d42b8, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc0008c1180, 0x1ff1aa0, 0xc000b16080, 0x5, 0xc000b16080, 0xc000d9aa40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0008c1180, 0x0, 0x0, 0xc000d9ae88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc0008c1180, 0xc0045da000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0021bca20, 0xc0045da000, 0x1000, 0x1000, 0xc000d9adc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc004b886c0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc004b886c0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0021bca20)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1150950 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0021bcd80)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1119412 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cd41cb0, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000792398, 0x72, 0xcd00, 0xcd92, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000792380, 0xc000f1e000, 0xcd92, 0xcd92, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000792380, 0xc000f1e000, 0xcd92, 0xcd92, 0xcd8d, 0xc000f1e000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0023dfe70, 0xc000f1e000, 0xcd92, 0xcd92, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0034d4378, 0xc000f1e000, 0xcd92, 0xcd92, 0xc002a5e9f8, 0xc000100000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc0009ac978, 0x1fed400, 0xc0034d4378, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc0009ac700, 0x1ff1aa0, 0xc0023dfe70, 0x5, 0xc0023dfe70, 0xc002a5ea40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0009ac700, 0x0, 0x0, 0xc002a5ee88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc0009ac700, 0xc0048c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc000f97200, 0xc0048c1000, 0x1000, 0x1000, 0xc002a5edc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0048bfec0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0048bfec0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc000f97200)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1174722 [IO wait]:
internal/poll.runtime_pollWait(0x7fb90c7a0af0, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000376218, 0x72, 0x0, 0x1, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000376200, 0xc005bfc941, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000376200, 0xc005bfc941, 0x1, 0x1, 0xc000632b10, 0xc0002b90e0, 0x466860)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0053a0000, 0xc005bfc941, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
net/http.(*connReader).backgroundRead(0xc005bfc930)
/usr/local/go/src/net/http/server.go:672 +0x58
created by net/http.(*connReader).startBackgroundRead
/usr/local/go/src/net/http/server.go:668 +0xd5
goroutine 1119347 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cd41bc8, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0005da718, 0x72, 0x900, 0x951, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0005da700, 0xc000478000, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0005da700, 0xc000478000, 0x951, 0x951, 0x20300000000000, 0x400, 0x400)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0035f5b40, 0xc000478000, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0034d41f8, 0xc000478000, 0x951, 0x951, 0xc0022a89f8, 0xc000708800, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000bb9af8, 0x1fed400, 0xc0034d41f8, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000bb9880, 0x1ff1aa0, 0xc0035f5b40, 0x5, 0xc0035f5b40, 0xc0022a8a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000bb9880, 0x0, 0x0, 0xc0022a8e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000bb9880, 0xc004544000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc003d7c480, 0xc004544000, 0x1000, 0x1000, 0xc0022a8dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0057a32c0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0057a32c0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc003d7c480)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1150979 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0017fd200)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1119348 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc003d7c480)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1026295 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90c792440, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0052de698, 0x72, 0xcd00, 0xcd92, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0052de680, 0xc000a12000, 0xcd92, 0xcd92, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0052de680, 0xc000a12000, 0xcd92, 0xcd92, 0x20300000000000, 0x7fb90d071fff, 0x400)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00056ed28, 0xc000a12000, 0xcd92, 0xcd92, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312cf0, 0xc000a12000, 0xcd92, 0xcd92, 0xc00165e9f8, 0xc000100000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc005416278, 0x1fed400, 0xc003312cf0, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc005416000, 0x1ff1aa0, 0xc00056ed28, 0x5, 0xc00056ed28, 0xc00165ea40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc005416000, 0x0, 0x0, 0xc00165ee88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc005416000, 0xc005424000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0053ae000, 0xc005424000, 0x1000, 0x1000, 0xc00165edc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0066efe60)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0066efe60, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0053ae000)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1150815 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90ccd7a58, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0007e6498, 0x72, 0xcd00, 0xcd94, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0007e6480, 0xc0021d6000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0007e6480, 0xc0021d6000, 0xcd94, 0xcd94, 0xcd8f, 0xc0021d6000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0007309e0, 0xc0021d6000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312e70, 0xc0021d6000, 0xcd94, 0xcd94, 0xc0007e29f8, 0xc000650400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000335af8, 0x1fed400, 0xc003312e70, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000335880, 0x1ff1aa0, 0xc0007309e0, 0x5, 0xc0007309e0, 0xc0007e2a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000335880, 0x0, 0x0, 0xc0007e2e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000335880, 0xc0042ea000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0015e2120, 0xc0042ea000, 0x1000, 0x1000, 0xc0007e2dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0047e6840)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0047e6840, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0015e2120)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1119413 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc000f97200)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1026296 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0053ae000)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1119351 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90c7926f8, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000130618, 0x72, 0x900, 0x951, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000130600, 0xc004546a80, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000130600, 0xc004546a80, 0x951, 0x951, 0x94c, 0xc004546a80, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc003614f90, 0xc004546a80, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc00094dc80, 0xc004546a80, 0x951, 0x951, 0xc0002aa9f8, 0xc000650400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc0022ae978, 0x1fed400, 0xc00094dc80, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc0022ae700, 0x1ff1aa0, 0xc003614f90, 0x5, 0xc003614f90, 0xc0002aaa40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0022ae700, 0x0, 0x0, 0xc0002aae88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc0022ae700, 0xc004562000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc00048d440, 0xc004562000, 0x1000, 0x1000, 0xc0002aadc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0045603c0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0045603c0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc00048d440)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1119280 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cca7610, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000376a98, 0x72, 0xcd00, 0xcd93, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000376a80, 0xc001c22000, 0xcd93, 0xcd93, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000376a80, 0xc001c22000, 0xcd93, 0xcd93, 0xcd8e, 0xc001c22000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc002ed6bb0, 0xc001c22000, 0xcd93, 0xcd93, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312c60, 0xc001c22000, 0xcd93, 0xcd93, 0xc000c729f8, 0xc000100000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc004898278, 0x1fed400, 0xc003312c60, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc004898000, 0x1ff1aa0, 0xc002ed6bb0, 0x5, 0xc002ed6bb0, 0xc000c72a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc004898000, 0x0, 0x0, 0xc000c72e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc004898000, 0xc004891000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0043b10e0, 0xc004891000, 0x1000, 0x1000, 0xc000c72dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0066231a0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0066231a0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0043b10e0)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1119349 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90c7a0f78, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc005fe0698, 0x72, 0xcd00, 0xcd94, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005fe0680, 0xc002f3a000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc005fe0680, 0xc002f3a000, 0xcd94, 0xcd94, 0xcd8f, 0xc002f3a000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000b17cb0, 0xc002f3a000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312cc0, 0xc002f3a000, 0xcd94, 0xcd94, 0xc0035cd9f8, 0xc000650400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc0008c0cf8, 0x1fed400, 0xc003312cc0, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc0008c0a80, 0x1ff1aa0, 0xc000b17cb0, 0x5, 0xc000b17cb0, 0xc0035cda40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0008c0a80, 0x0, 0x0, 0xc0035cde88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc0008c0a80, 0xc004556000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc000de3560, 0xc004556000, 0x1000, 0x1000, 0xc0035cddc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc006aa4cc0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc006aa4cc0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc000de3560)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 995331 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cca7440, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc005666598, 0x72, 0xcd00, 0xcd94, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005666580, 0xc002cc8000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc005666580, 0xc002cc8000, 0xcd94, 0xcd94, 0x20300000000000, 0x7fb90d071fff, 0x400)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000730058, 0xc002cc8000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0034d42e8, 0xc002cc8000, 0xcd94, 0xcd94, 0xc001f6c9f8, 0xc0004f6400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000b46cf8, 0x1fed400, 0xc0034d42e8, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000b46a80, 0x1ff1aa0, 0xc000730058, 0x5, 0xc000730058, 0xc001f6ca40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000b46a80, 0x0, 0x0, 0xc001f6ce88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000b46a80, 0xc00615e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc005267200, 0xc00615e000, 0x1000, 0x1000, 0xc001f6cdc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc00615c360)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc00615c360, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc005267200)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 995332 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc005267200)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1149511 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90c7929b0, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc004cb8598, 0x72, 0xcd00, 0xcd94, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc004cb8580, 0xc0030e4000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc004cb8580, 0xc0030e4000, 0xcd94, 0xcd94, 0x20300000000000, 0x7fb90d071fff, 0x400)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0005d8698, 0xc0030e4000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312c30, 0xc0030e4000, 0xcd94, 0xcd94, 0xc0043ee9f8, 0xc000708800, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc004af0278, 0x1fed400, 0xc003312c30, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc004af0000, 0x1ff1aa0, 0xc0005d8698, 0x5, 0xc0005d8698, 0xc0043eea40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc004af0000, 0x0, 0x0, 0xc0043eee88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc004af0000, 0xc00479c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0043b0120, 0xc00479c000, 0x1000, 0x1000, 0xc0043eedc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0061a4c00)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0061a4c00, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0043b0120)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1095135 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90c792c68, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000130918, 0x72, 0x4b00, 0x4b66, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000130900, 0xc002d29000, 0x4b66, 0x4b66, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000130900, 0xc002d29000, 0x4b66, 0x4b66, 0x20300000000000, 0x7fb90d071fff, 0x400)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc004e58000, 0xc002d29000, 0x4b66, 0x4b66, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc00094db48, 0xc002d29000, 0x4b66, 0x4b66, 0xc0010b39f8, 0xc000100000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000a38278, 0x1fed400, 0xc00094db48, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000a38000, 0x1ff1aa0, 0xc004e58000, 0x5, 0xc004e58000, 0xc0010b3a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000a38000, 0x0, 0x0, 0xc0010b3e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000a38000, 0xc0047d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc002d6a480, 0xc0047d0000, 0x1000, 0x1000, 0xc0010b3dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc006cba720)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc006cba720, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc002d6a480)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1119302 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cca7358, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000048318, 0x72, 0x900, 0x952, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000048300, 0xc00342ea80, 0x952, 0x952, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000048300, 0xc00342ea80, 0x952, 0x952, 0x94d, 0xc00342ea80, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000fb6000, 0xc00342ea80, 0x952, 0x952, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0034d4318, 0xc00342ea80, 0x952, 0x952, 0xc0003bd9f8, 0xc000650400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc0003345f8, 0x1fed400, 0xc0034d4318, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000334380, 0x1ff1aa0, 0xc000fb6000, 0x5, 0xc000fb6000, 0xc0003bda40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000334380, 0x0, 0x0, 0xc0003bde88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000334380, 0xc003464000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0006e9200, 0xc003464000, 0x1000, 0x1000, 0xc0003bddc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc004beb6e0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc004beb6e0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0006e9200)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1151061 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0043b0360)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1119277 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cca78c8, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0005da818, 0x72, 0x900, 0x951, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0005da800, 0xc006d20000, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0005da800, 0xc006d20000, 0x951, 0x951, 0x20300000000000, 0x400, 0x400)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0005d9900, 0xc006d20000, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003cbfe00, 0xc006d20000, 0x951, 0x951, 0xc0057ea9f8, 0xc000100000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc005a65078, 0x1fed400, 0xc003cbfe00, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc005a64e00, 0x1ff1aa0, 0xc0005d9900, 0x5, 0xc0005d9900, 0xc0057eaa40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc005a64e00, 0x0, 0x0, 0xc0057eae88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc005a64e00, 0xc00415b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0006e9320, 0xc00415b000, 0x1000, 0x1000, 0xc0057eadc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc002e3ae40)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc002e3ae40, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0006e9320)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1119363 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90ccd7ee0, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc005fe0498, 0x72, 0xcd00, 0xcd94, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005fe0480, 0xc000cb6000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc005fe0480, 0xc000cb6000, 0xcd94, 0xcd94, 0x20300000000000, 0x400, 0x400)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc002164000, 0xc000cb6000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312d20, 0xc000cb6000, 0xcd94, 0xcd94, 0xc000faa9f8, 0xc000100000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000205078, 0x1fed400, 0xc003312d20, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000204e00, 0x1ff1aa0, 0xc002164000, 0x5, 0xc002164000, 0xc000faaa40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000204e00, 0x0, 0x0, 0xc000faae88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000204e00, 0xc0032f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc000f97440, 0xc0032f3000, 0x1000, 0x1000, 0xc000faadc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc000f99560)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc000f99560, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc000f97440)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1087742 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cd42138, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000130818, 0x72, 0xcd00, 0xcd94, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000130800, 0xc00127a000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000130800, 0xc00127a000, 0xcd94, 0xcd94, 0xcd8f, 0xc00127a000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000c9c050, 0xc00127a000, 0xcd94, 0xcd94, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312de0, 0xc00127a000, 0xcd94, 0xcd94, 0xc00454d9f8, 0xc000100000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc005a64978, 0x1fed400, 0xc003312de0, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc005a64700, 0x1ff1aa0, 0xc000c9c050, 0x5, 0xc000c9c050, 0xc00454da40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc005a64700, 0x0, 0x0, 0xc00454de88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc005a64700, 0xc004816000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc001933320, 0xc004816000, 0x1000, 0x1000, 0xc00454ddc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc00352b860)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc00352b860, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc001933320)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1150948 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0016bb560)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1076625 [select, 16 minutes]:
main.downsampleBucket.func4(0x0, 0x0)
/app/cmd/thanos/downsample.go:297 +0x25e
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0021d4840, 0xc002b04000)
/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x66
goroutine 1150844 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0021bca20)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1150798 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90ccd8198, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0007e6598, 0x72, 0xcd00, 0xcd93, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0007e6580, 0xc001bc8000, 0xcd93, 0xcd93, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0007e6580, 0xc001bc8000, 0xcd93, 0xcd93, 0xcd8e, 0xc001bc8000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc004e594d0, 0xc001bc8000, 0xcd93, 0xcd93, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc00094dc20, 0xc001bc8000, 0xcd93, 0xcd93, 0xc001df79f8, 0xc000708800, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000b7de78, 0x1fed400, 0xc00094dc20, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000b7dc00, 0x1ff1aa0, 0xc004e594d0, 0x5, 0xc004e594d0, 0xc001df7a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000b7dc00, 0x0, 0x0, 0xc001df7e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000b7dc00, 0xc0044e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc000f96000, 0xc0044e5000, 0x1000, 0x1000, 0xc001df7dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc005c5ac00)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc005c5ac00, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc000f96000)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1150879 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cca6fb8, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc004f4a798, 0x72, 0x900, 0x952, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc004f4a780, 0xc00468a000, 0x952, 0x952, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc004f4a780, 0xc00468a000, 0x952, 0x952, 0x94d, 0xc00468a000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0007309d0, 0xc00468a000, 0x952, 0x952, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0034d4258, 0xc00468a000, 0x952, 0x952, 0xc00178f9f8, 0xc0004f6400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000334978, 0x1fed400, 0xc0034d4258, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000334700, 0x1ff1aa0, 0xc0007309d0, 0x5, 0xc0007309d0, 0xc00178fa40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000334700, 0x0, 0x0, 0xc00178fe88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000334700, 0xc0046a4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0066f7d40, 0xc0046a4000, 0x1000, 0x1000, 0xc00178fdc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0057a2060)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0057a2060, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0066f7d40)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1150964 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0043b0480)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1151013 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0016bb9e0)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1119304 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90c792270, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc005fe0398, 0x72, 0xcd00, 0xcd92, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005fe0380, 0xc001550000, 0xcd92, 0xcd92, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc005fe0380, 0xc001550000, 0xcd92, 0xcd92, 0xcd8d, 0xc001550000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000c6a100, 0xc001550000, 0xcd92, 0xcd92, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc0034d43a8, 0xc001550000, 0xcd92, 0xcd92, 0xc006db79f8, 0xc000650400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000b38978, 0x1fed400, 0xc0034d43a8, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000b38700, 0x1ff1aa0, 0xc000c6a100, 0x5, 0xc000c6a100, 0xc006db7a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000b38700, 0x0, 0x0, 0xc006db7e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000b38700, 0xc003478000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc00172b320, 0xc003478000, 0x1000, 0x1000, 0xc006db7dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc003467140)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc003467140, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc00172b320)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 952095 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0006ac7e0)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1150980 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90c7a1400, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc004f4a598, 0x72, 0x900, 0x951, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc004f4a580, 0xc0047e2000, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc004f4a580, 0xc0047e2000, 0x951, 0x951, 0x94c, 0xc0047e2000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000af4a58, 0xc0047e2000, 0x951, 0x951, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc00094db78, 0xc0047e2000, 0x951, 0x951, 0xc00379f9f8, 0xc000708800, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc000248278, 0x1fed400, 0xc00094db78, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc000248000, 0x1ff1aa0, 0xc000af4a58, 0x5, 0xc000af4a58, 0xc00379fa40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc000248000, 0x0, 0x0, 0xc00379fe88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc000248000, 0xc0047f4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0006ac000, 0xc0047f4000, 0x1000, 0x1000, 0xc00379fdc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0062ea9c0)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0062ea9c0, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0006ac000)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1150978 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cca70a0, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc004cb8798, 0x72, 0x4b00, 0x4b67, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc004cb8780, 0xc0030d0000, 0x4b67, 0x4b67, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc004cb8780, 0xc0030d0000, 0x4b67, 0x4b67, 0x4b62, 0xc0030d0000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00000fdc0, 0xc0030d0000, 0x4b67, 0x4b67, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312e40, 0xc0030d0000, 0x4b67, 0x4b67, 0xc0010ae9f8, 0xc000650400, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc0048985f8, 0x1fed400, 0xc003312e40, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc004898380, 0x1ff1aa0, 0xc00000fdc0, 0x5, 0xc00000fdc0, 0xc0010aea40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc004898380, 0x0, 0x0, 0xc0010aee88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc004898380, 0xc0047cc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0017fd200, 0xc0047cc000, 0x1000, 0x1000, 0xc0010aedc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc00009af60)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc00009af60, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0017fd200)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1150799 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc000f96000)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1150880 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0066f7d40)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
goroutine 1151027 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fb90cca6c18, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0007e6698, 0x72, 0x900, 0x950, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0007e6680, 0xc004478000, 0x950, 0x950, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0007e6680, 0xc004478000, 0x950, 0x950, 0x94b, 0xc004478000, 0x5)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0005d86a0, 0xc004478000, 0x950, 0x950, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc003312db0, 0xc004478000, 0x950, 0x950, 0xc0003c09f8, 0xc000708800, 0x0)
/usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc004af0978, 0x1fed400, 0xc003312db0, 0x40b925, 0x1a877e0, 0x1c813e0)
/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc004af0700, 0x1ff1aa0, 0xc0005d86a0, 0x5, 0xc0005d86a0, 0xc0003c0a40)
/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc004af0700, 0x0, 0x0, 0xc0003c0e88)
/usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc004af0700, 0xc00447e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0016bb7a0, 0xc00447e000, 0x1000, 0x1000, 0xc0003c0dc4, 0x2, 0x2)
/usr/local/go/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0063bec60)
/usr/local/go/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0063bec60, 0x1, 0x0, 0x1, 0x1, 0x1, 0x0)
/usr/local/go/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0016bb7a0)
/usr/local/go/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1743 +0xc77
goroutine 1119364 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc000f97440)
/usr/local/go/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1744 +0xc9c
Anything else we need to know:
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 22 (19 by maintainers)
I’ll be testing today with v0.21.0 and report back. It might be taking a while, I’ve observed the bug when the “big” downsampling work comes around every other week or so, I’ll report back!
re: server side, it could be certainly a problem there too, however only
meta.jsonhas been downloaded and I can stop envoy and thanos-compact downsampling never recovers.On the server side (swift + s3 api compat) I can see the index for that block being downloaded, though at one of the multipart segment (segment
69) swift logs a 499 status which indicates the client disconnecting. Another piece of the puzzle is envoy which does the TLS termination for swift, although I can’t find any error or other signal in its logs.