moby: fatal error: concurrent map writes (buildkit/cache/remotecache/v1.(*item).LinkFrom)

Description

Running multiple builds simultaneously on one docker daemon with extensive use of --cache-from and multistage builds (K8s environment, dedicated dind dockers for CI purpose) got this (latest from the logs):

fatal error: concurrent map writes

goroutine 2330884 [running]:
runtime.throw(0x2057f4f, 0x15)
	/usr/local/go/src/runtime/panic.go:774 +0x74 fp=0xc00403a018 sp=0xc004039fe8 pc=0x431ab4
runtime.mapassign_fast64ptr(0x2cbfa00, 0xc00774e990, 0xc01158aae0, 0xc00e8556b8)
	/usr/local/go/src/runtime/map_fast64.go:191 +0x36b fp=0xc00403a058 sp=0xc00403a018 pc=0x41423b
github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.(*item).LinkFrom(0xc01158aae0, 0x31920c0, 0xc009223080, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/chains.go:142 +0x1b4 fp=0xc00403a0d0 sp=0xc00403a058 pc=0x1c9e724
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.addBacklinks.func1(0xc0250e73c0, 0x19, 0x0, 0x0, 0xc013094a00, 0x47, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:42 +0x117 fp=0xc00403a168 sp=0xc00403a0d0 pc=0x189bd57
github.com/docker/docker/vendor/github.com/moby/buildkit/solver/bboltcachestorage.(*Store).WalkBacklinks(0xc000010338, 0xc0046ddc40, 0x19, 0xc00f7333b0, 0xc003c14558, 0x411199)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/bboltcachestorage/storage.go:440 +0x1dd fp=0xc00403a260 sp=0xc00403a168 pc=0x1d6a56d
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.addBacklinks(0x7f6f443409a0, 0xc008c0eb80, 0x31920c0, 0xc01158aae0, 0xc00045e0a0, 0xc0046ddc40, 0x19, 0xc0153196b0, 0xc0110b06e0, 0xc00c9d9820, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:29 +0x159 fp=0xc00403a2b0 sp=0xc00403a260 pc=0x1886769
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc0075223c0, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0xc000f18470, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:164 +0x446 fp=0xc00403a5b0 sp=0xc00403a2b0 pc=0x1886e26
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc007522af0, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0xc000df2000, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403a8b0 sp=0xc00403a5b0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc029715e00, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0xc0110b06e0, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403abb0 sp=0xc00403a8b0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc0121a0dc0, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0xc0110b06e0, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403aeb0 sp=0xc00403abb0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc029194230, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0xc0110b06e0, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403b1b0 sp=0xc00403aeb0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc0291943c0, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0x6, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403b4b0 sp=0xc00403b1b0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc029194550, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0x6, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403b7b0 sp=0xc00403b4b0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc029194730, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0x6, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403bab0 sp=0xc00403b7b0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc029194960, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0x6, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403bdb0 sp=0xc00403bab0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc029194be0, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0x6, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403c0b0 sp=0xc00403bdb0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc029195360, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0x6, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403c3b0 sp=0xc00403c0b0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc029195770, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0x6, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403c6b0 sp=0xc00403c3b0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc029195b30, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0x6, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403c9b0 sp=0xc00403c6b0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc029195db0, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0x6, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403ccb0 sp=0xc00403c9b0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc018c32190, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0xc000df2000, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403cfb0 sp=0xc00403ccb0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc018c325a0, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0xc0110b06e0, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403d2b0 sp=0xc00403cfb0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc007ebb770, 0x31bff80, 0xc0153196e0, 0x7f6f443409a0, 0xc008c0eb80, 0xc0110b0720, 0x2, 0x315bda0, 0xc0110b06e0, 0x0, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab fp=0xc00403d5b0 sp=0xc00403d2b0 pc=0x188768b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.inlineCache(0x31bfec0, 0xc01ad69180, 0x31c0a00, 0xc008c0eb80, 0x31c7b00, 0xc010cb0aa0, 0x315bda0, 0xc0110b06e0, 0x0, 0x0, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:287 +0x350 fp=0xc00403d758 sp=0xc00403d5b0 pc=0x1ce9190
github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0xc000e42a20, 0x31bfec0, 0xc01ad69180, 0xc016a90b01, 0x19, 0xc004a5c4a4, 0x19, 0x0, 0x0, 0x2045d85, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:176 +0x15f2 fp=0xc00403dc70 sp=0xc00403d758 pc=0x1ce8482
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Solve(0xc000836360, 0x31bfec0, 0xc01ad69180, 0xc020903320, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:275 +0x4fc fp=0xc00403def0 sp=0xc00403dc70 pc=0x1cf256c
github.com/docker/docker/builder/builder-next.(*Builder).Build.func2(0xc02a723768, 0x0)
	/go/src/github.com/docker/docker/builder/builder-next/builder.go:376 +0x76 fp=0xc00403df58 sp=0xc00403def0 pc=0x1d92b76
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc004a98e70, 0xc01587e0f0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66 fp=0xc00403dfd0 sp=0xc00403df58 pc=0x6e21f6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00403dfd8 sp=0xc00403dfd0 pc=0x462331
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 1 [chan receive, 3549 minutes]:
main.(*DaemonCli).start(0xc000a93320, 0xc000120a20, 0x0, 0x0)
	/go/src/github.com/docker/docker/cmd/dockerd/daemon.go:249 +0xc04
main.runDaemon(...)
	/go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:13
main.newDaemonCommand.func1(0xc000473080, 0xc0009d86e0, 0x0, 0xa, 0x0, 0x0)
	/go/src/github.com/docker/docker/cmd/dockerd/docker.go:34 +0x7c
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc000473080, 0xc000122010, 0xa, 0xa, 0xc000473080, 0xc000122010)
	/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:850 +0x462
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000473080, 0x0, 0x0, 0x7)
	/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:958 +0x34b
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:895
main.main()
	/go/src/github.com/docker/docker/cmd/dockerd/docker.go:97 +0x191

goroutine 19 [syscall, 855 minutes]:
os/signal.signal_recv(0x3197a00)
	/usr/local/go/src/runtime/sigqueue.go:147 +0x9e
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x24
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x43

goroutine 57 [select, 3550 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream.func5(0xc000aad500, 0xc000b88000, 0x31bff80, 0xc000b7e180)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:319 +0xd9
created by github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:318 +0x9a7

goroutine 72 [select]:
github.com/docker/docker/vendor/go.opencensus.io/stats/view.(*worker).start(0xc00024e460)
	/go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:154 +0x102
created by github.com/docker/docker/vendor/go.opencensus.io/stats/view.init.0
	/go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:32 +0x59

goroutine 80 [IO wait, 3550 minutes]:
internal/poll.runtime_pollWait(0x7f6f44326b90, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000156298, 0x72, 0x0, 0x0, 0x20390e1)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000156280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc000156280, 0xc00069c000, 0x7f6f48737d98, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc000a79a10, 0xc0008d9d98, 0x40ed1a, 0x30)
	/usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc000a79a10, 0x2f22f80, 0xc00093c660, 0x2cf5dc0, 0x44fd140)
	/usr/local/go/src/net/unixsock.go:260 +0x49
net/http.(*Server).Serve(0xc0009d41c0, 0x31b0100, 0xc000a79a10, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2925 +0x282
net/http.Serve(...)
	/usr/local/go/src/net/http/server.go:2497
github.com/docker/docker/daemon.(*Daemon).listenMetricsSock.func1(0x31b0100, 0xc000a79a10, 0xc000446180)
	/go/src/github.com/docker/docker/daemon/metrics_unix.go:33 +0x101
created by github.com/docker/docker/daemon.(*Daemon).listenMetricsSock
	/go/src/github.com/docker/docker/daemon/metrics_unix.go:31 +0x1af

goroutine 77 [chan receive, 3550 minutes]:
github.com/docker/docker/daemon.(*Daemon).setupDumpStackTrap.func1(0xc000395b60, 0x2048eea, 0xf)
	/go/src/github.com/docker/docker/daemon/debugtrap_unix.go:18 +0x43
created by github.com/docker/docker/daemon.(*Daemon).setupDumpStackTrap
	/go/src/github.com/docker/docker/daemon/debugtrap_unix.go:17 +0xc1

goroutine 88 [select, 3550 minutes]:
github.com/docker/docker/libcontainerd/supervisor.(*remote).monitorDaemon(0xc0003ee800, 0x31bfec0, 0xc0000e1ec0)
	/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:318 +0xe1d
created by github.com/docker/docker/libcontainerd/supervisor.Start
	/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:90 +0x426

goroutine 47 [syscall, 3550 minutes]:
syscall.Syscall6(0xf7, 0x1, 0x2a, 0xc000271c90, 0x1000004, 0x0, 0x0, 0xc000271d10, 0x5ed969, 0xc000269340)
	/usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
os.(*Process).blockUntilWaitable(0xc000866390, 0xc00025e080, 0x7e, 0xc00014bd98)
	/usr/local/go/src/os/wait_waitid.go:31 +0x9a
os.(*Process).wait(0xc000866390, 0x11, 0x3, 0xc00025a960)
	/usr/local/go/src/os/exec_unix.go:22 +0x3b
os.(*Process).Wait(...)
	/usr/local/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc000128160, 0x6, 0xc000271e60)
	/usr/local/go/src/os/exec/exec.go:506 +0x62
github.com/docker/docker/libcontainerd/supervisor.(*remote).startContainerd.func1(0xc000128160, 0xc0003ee800)
	/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:214 +0x45
created by github.com/docker/docker/libcontainerd/supervisor.(*remote).startContainerd
	/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:212 +0x2b7

goroutine 48 [select, 3550 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000140e40)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc4
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x16f

goroutine 49 [chan receive, 3550 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc0001f22c0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x6ec
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x12a

goroutine 103 [IO wait, 71 minutes]:
internal/poll.runtime_pollWait(0x7f6f44326c60, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0008fe518, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0008fe500, 0xc000902000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc0008fe500, 0xc000902000, 0x8000, 0x8000, 0x0, 0x800000601, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc00011e020, 0xc000902000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
bufio.(*Reader).Read(0xc00056c480, 0xc00090e038, 0x9, 0x9, 0xc0008e1de0, 0x4070ee, 0xc0008fa420)
	/usr/local/go/src/bufio/bufio.go:226 +0x26c
io.ReadAtLeast(0x3159480, 0xc00056c480, 0xc00090e038, 0x9, 0x9, 0x9, 0x45bfe0, 0xc0008fa420, 0xc0008e1e18)
	/usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc00090e038, 0x9, 0x9, 0x3159480, 0xc00056c480, 0x0, 0xc000000000, 0x10000000040ed1a, 0xc0001445b0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00090e000, 0xc00404e480, 0xc00404e480, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0001b4380)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x185
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd25

goroutine 104 [select, 71 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0001445a0, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x124
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00056c600, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0001b4380)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7d
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xedd

goroutine 115 [chan receive, 855 minutes]:
github.com/docker/docker/pkg/signal.Trap.func1(0xc00056c960, 0x315c480, 0xc00013fe30, 0xc000aa05e0)
	/go/src/github.com/docker/docker/pkg/signal/trap.go:38 +0x5d
created by github.com/docker/docker/pkg/signal.Trap
	/go/src/github.com/docker/docker/pkg/signal/trap.go:36 +0x11d

goroutine 81 [select, 3550 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0004463c0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc4
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x16f

goroutine 130 [chan receive, 3550 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc00012b080)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x6ec
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x12a

goroutine 132 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f6f44326ac0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000156498, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000156480, 0xc00096e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc000156480, 0xc00096e000, 0x8000, 0x8000, 0x0, 0x800010601, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc0006806b0, 0xc00096e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
bufio.(*Reader).Read(0xc0001d92c0, 0xc00096a118, 0x9, 0x9, 0xc0000bb180, 0x7f6f48737d98, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x26c
io.ReadAtLeast(0x3159480, 0xc0001d92c0, 0xc00096a118, 0x9, 0x9, 0x9, 0xb36af7, 0xc00ac66bec, 0xc0008d6de8)
	/usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc00096a118, 0x9, 0x9, 0x3159480, 0xc0001d92c0, 0x0, 0xc0239ba900000000, 0xc1b8a24be94a, 0x45a58c0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00096a0e0, 0xc00ac66be0, 0xc00ac66be0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00097a000)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x185
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd25

goroutine 133 [select, 1 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00024ec30, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x124
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001d93e0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00097a000)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7d
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xedd

goroutine 116 [select, 3550 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000141380)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc4
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x16f

goroutine 117 [chan receive, 3550 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc0001f2580)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x6ec
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x12a

goroutine 119 [IO wait, 71 minutes]:
internal/poll.runtime_pollWait(0x7f6f443269f0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000848b98, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000848b80, 0xc00099c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc000848b80, 0xc00099c000, 0x8000, 0x8000, 0x0, 0x800000601, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc000010200, 0xc00099c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
bufio.(*Reader).Read(0xc0002021e0, 0xc0009d43b8, 0x9, 0x9, 0xc0008d7de0, 0x4070ee, 0xc00024d740)
	/usr/local/go/src/bufio/bufio.go:226 +0x26c
io.ReadAtLeast(0x3159480, 0xc0002021e0, 0xc0009d43b8, 0x9, 0x9, 0x9, 0x45bfe0, 0xc00024d740, 0xc0008d7e18)
	/usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc0009d43b8, 0x9, 0x9, 0x3159480, 0xc0002021e0, 0x0, 0xc000000000, 0x10000000040ed1a, 0xc0001f5640)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0009d4380, 0xc00404e4a0, 0xc00404e4a0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00023c8c0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x185
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd25

goroutine 120 [select, 71 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0001f5630, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x124
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000202300, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00023c8c0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7d
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xedd

goroutine 106 [select, 3550 minutes]:
github.com/docker/docker/libcontainerd/remote.(*client).processEventStream(0xc00092e2a0, 0x31bfec0, 0xc0000e1ec0, 0x204419d, 0xc)
	/go/src/github.com/docker/docker/libcontainerd/remote/client.go:782 +0x314
created by github.com/docker/docker/libcontainerd/remote.NewClient
	/go/src/github.com/docker/docker/libcontainerd/remote/client.go:70 +0x27f

goroutine 134 [select, 3550 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream.func5(0xc00074ae00, 0xc00048ea20, 0x31bff80, 0xc000978840)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:319 +0xd9
created by github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:318 +0x9a7

goroutine 135 [select, 3550 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc00005c700)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:318 +0xce
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:333
github.com/docker/docker/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc000156800, 0x3019dc0, 0xc0001d9800, 0x0, 0x20385ce, 0x7)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:871 +0x755
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000156800, 0x203000, 0x203000)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:736 +0x48
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc00048ea20, 0xc0000c5e48, 0xc0000c5e18, 0xc0002438c0, 0x48)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:594 +0x9e
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc00048ea20, 0x3019dc0, 0xc0001d9800, 0x43084c, 0xc000202300)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:735 +0x105
github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1.(*eventsSubscribeClient).Recv(0xc00096c4f0, 0x0, 0xc00024d560, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1/events.pb.go:354 +0x64
github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe.func1(0xc0001d96e0, 0x31e09c0, 0xc00096c4f0, 0xc0000f9320, 0x31bfec0, 0xc000446700)
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:99 +0x8f
created by github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:95 +0x1c6

goroutine 107 [sync.Cond.Wait, 3550 minutes]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc00092a890, 0x0)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc00092a880)
	/usr/local/go/src/sync/cond.go:56 +0x9f
github.com/docker/docker/daemon/stats.(*Collector).Run(0xc00092a840)
	/go/src/github.com/docker/docker/daemon/stats/collector.go:95 +0x4a
created by github.com/docker/docker/daemon.(*Daemon).newStatsCollector
	/go/src/github.com/docker/docker/daemon/stats_collector.go:24 +0x7f

goroutine 108 [chan receive, 5 minutes]:
github.com/docker/docker/daemon.(*Daemon).execCommandGC(0xc00000c1e0)
	/go/src/github.com/docker/docker/daemon/exec.go:309 +0x14b
created by github.com/docker/docker/daemon.NewDaemon
	/go/src/github.com/docker/docker/daemon/daemon.go:1140 +0x2aa9

goroutine 109 [select, 17 minutes]:
github.com/docker/docker/libcontainerd/remote.(*client).processEventStream(0xc00092e690, 0x31bfec0, 0xc0000e1ec0, 0x203480d, 0x4)
	/go/src/github.com/docker/docker/libcontainerd/remote/client.go:782 +0x314
created by github.com/docker/docker/libcontainerd/remote.NewClient
	/go/src/github.com/docker/docker/libcontainerd/remote/client.go:70 +0x27f

goroutine 110 [select]:
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).watchLoop(0xc000936100)
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/store.go:430 +0xc8
created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).startWatch
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/store.go:447 +0xcf

goroutine 58 [select, 17 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc000b9a000, 0xc00033e1b0, 0x5, 0x5, 0x60, 0x0, 0x3115bc8)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:186 +0xd5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000b9a000, 0xc00033e1b0, 0x5, 0x5, 0xc000b71ef0, 0xc000b9c000, 0xc000b71990)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:166 +0x197
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000b7e210, 0xc00033e1b0, 0x5, 0x5, 0xc02383e0e5, 0xc000b719b8, 0xb7a78e)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:479 +0x57
io.ReadAtLeast(0x315d520, 0xc000b7e210, 0xc00033e1b0, 0x5, 0x5, 0x5, 0x0, 0x0, 0xc000b7a1e0)
	/usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000b98000, 0xc00033e1b0, 0x5, 0x5, 0x25, 0x2d63100, 0x3019dc0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:463 +0xcf
github.com/docker/docker/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc00033e1a0, 0x1000000, 0xc0089940c0, 0xc02383e0a0, 0x99, 0x99, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/rpc_util.go:508 +0x65
github.com/docker/docker/vendor/google.golang.org/grpc.recvAndDecompress(0xc00033e1a0, 0xc000b98000, 0x0, 0x0, 0x1000000, 0x0, 0x0, 0x0, 0xc02383e0a0, 0x99, ...)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4f
github.com/docker/docker/vendor/google.golang.org/grpc.recv(0xc00033e1a0, 0x7f6f444d4280, 0x45cb9e0, 0xc000b98000, 0x0, 0x0, 0x3019dc0, 0xc002d5b2c0, 0x1000000, 0x0, ...)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/rpc_util.go:707 +0x9d
github.com/docker/docker/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc000b8a000, 0x3019dc0, 0xc002d5b2c0, 0x0, 0xc000b71d38, 0x45ee00)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:885 +0xef
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000b8a000, 0x203000, 0x203000)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:736 +0x48
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc000b88000, 0xc000b71e48, 0xc000b71e18, 0x0, 0x48)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:590 +0x362
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc000b88000, 0x3019dc0, 0xc002d5b2c0, 0xc000b71eec, 0x2)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:735 +0x105
github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1.(*eventsSubscribeClient).Recv(0xc000b740c0, 0xc000b71ee8, 0x2, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1/events.pb.go:354 +0x64
github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe.func1(0xc000b7c000, 0x31e09c0, 0xc000b740c0, 0xc000b7a000, 0x31bfec0, 0xc000b72000)
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:99 +0x8f
created by github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:95 +0x1c6

goroutine 173 [select, 3550 minutes]:
github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay.(*driver).peerOpRoutine(0xc000936200, 0x31bfec0, 0xc00092ba80, 0xc0008fb3e0)
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay/peerdb.go:279 +0x10c
created by github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay.Init
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay/overlay.go:77 +0x17f

goroutine 174 [IO wait]:
internal/poll.runtime_pollWait(0x7f6f44326920, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0008ff818, 0x72, 0x0, 0x0, 0x20390e1)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0008ff800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc0008ff800, 0xc0243dcce8, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc000bfaab0, 0xc00107ff48, 0xc00107ff50, 0x18)
	/usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc000bfaab0, 0x3110468, 0x31ea740, 0xc0243dcce8, 0xc000936100)
	/usr/local/go/src/net/unixsock.go:260 +0x49
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).acceptClientConnections(0xc000936100, 0xc000bfe150, 0x2c, 0x31b0100, 0xc000bfaab0)
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox_externalkey_unix.go:144 +0x3d
created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).startExternalKeyListener
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox_externalkey_unix.go:138 +0x339

goroutine 195 [chan receive, 3549 minutes]:
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*scheduler).loop.func2(0xc0002f8000)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:76 +0x3e
created by github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*scheduler).loop
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:75 +0x79

goroutine 123 [chan receive, 3549 minutes]:
main.(*DaemonCli).setupConfigReloadTrap.func1(0xc000e42de0, 0xc000a93320)
	/go/src/github.com/docker/docker/cmd/dockerd/daemon_unix.go:83 +0x57
created by main.(*DaemonCli).setupConfigReloadTrap
	/go/src/github.com/docker/docker/cmd/dockerd/daemon_unix.go:82 +0xb7

goroutine 124 [chan receive, 3549 minutes]:
github.com/docker/docker/api/server.(*Server).serveAPI(0xc00045e5f0, 0x406bc5, 0xc000c79aa0)
	/go/src/github.com/docker/docker/api/server/server.go:94 +0x14c
github.com/docker/docker/api/server.(*Server).Wait(0xc00045e5f0, 0xc0009bc3c0)
	/go/src/github.com/docker/docker/api/server/server.go:199 +0x31
created by main.(*DaemonCli).start
	/go/src/github.com/docker/docker/cmd/dockerd/daemon.go:242 +0xbd6

goroutine 126 [IO wait, 3549 minutes]:
internal/poll.runtime_pollWait(0x7f6f44326e00, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000846618, 0x72, 0x0, 0x0, 0x20390e1)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000846600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc000846600, 0xc000e0c000, 0x7f6f487398b8, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc000a93740, 0xc000985da0, 0x40ed1a, 0x30)
	/usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc000a93740, 0x2f22f80, 0xc000cc60f0, 0x2cf5dc0, 0x44fd140)
	/usr/local/go/src/net/unixsock.go:260 +0x49
net/http.(*Server).Serve(0xc00023a000, 0x31b0100, 0xc000a93740, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2925 +0x282
github.com/docker/docker/api/server.(*HTTPServer).Serve(...)
	/go/src/github.com/docker/docker/api/server/server.go:112
github.com/docker/docker/api/server.(*Server).serveAPI.func1(0xc00056c1e0, 0xc0001119a0)
	/go/src/github.com/docker/docker/api/server/server.go:86 +0xd1
created by github.com/docker/docker/api/server.(*Server).serveAPI
	/go/src/github.com/docker/docker/api/server/server.go:83 +0x8d

goroutine 127 [IO wait]:
internal/poll.runtime_pollWait(0x7f6f44326d30, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000846818, 0x72, 0x0, 0x0, 0x20390e1)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000846800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc000846800, 0xc0008d8da0, 0x859996, 0xc00023a180)
	/usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*TCPListener).accept(0xc000111a40, 0x60adf733, 0xc0008d8da0, 0x4cbee8)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x34
net.(*TCPListener).Accept(0xc000111a40, 0xc0008d8df0, 0x18, 0xc0006c1800, 0x858eb6)
	/usr/local/go/src/net/tcpsock.go:261 +0x49
net/http.(*Server).Serve(0xc00023a0e0, 0x31b00c0, 0xc000111a40, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2925 +0x282
github.com/docker/docker/api/server.(*HTTPServer).Serve(...)
	/go/src/github.com/docker/docker/api/server/server.go:112
github.com/docker/docker/api/server.(*Server).serveAPI.func1(0xc00056c1e0, 0xc000111b20)
	/go/src/github.com/docker/docker/api/server/server.go:86 +0xd1
created by github.com/docker/docker/api/server.(*Server).serveAPI
	/go/src/github.com/docker/docker/api/server/server.go:83 +0x8d

goroutine 2342219 [sleep]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
time.Sleep(0x2540be400)
	/usr/local/go/src/runtime/time.go:105 +0x15d
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*Job).Discard.func1(0xc0094b8f80)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/jobs.go:532 +0x36
created by github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*Job).Discard
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/jobs.go:530 +0x3b3

goroutine 226 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc00092a390, 0xc000001daa)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc00092a380)
	/usr/local/go/src/sync/cond.go:56 +0x9f
github.com/docker/docker/vendor/github.com/moby/buildkit/util/cond.(*StatefulCond).Wait(0xc000e3a810)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/cond/cond.go:28 +0xb1
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*scheduler).loop(0xc0002f8000)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:101 +0x18c
created by github.com/docker/docker/vendor/github.com/moby/buildkit/solver.newScheduler
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:35 +0x1ad

goroutine 228 [select, 3549 minutes]:
github.com/docker/docker/daemon.(*Daemon).ProcessClusterNotifications(0xc00000c1e0, 0x31bfec0, 0xc0000e1ec0, 0xc000203200)
	/go/src/github.com/docker/docker/daemon/events.go:126 +0xe7
created by main.(*DaemonCli).start
	/go/src/github.com/docker/docker/cmd/dockerd/daemon.go:234 +0xb71

goroutine 2331335 [IO wait]:
internal/poll.runtime_pollWait(0x7f6f44342340, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc003289818, 0x72, 0xf00, 0xf1a, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc003289800, 0xc009b38000, 0xf1a, 0xf1a, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc003289800, 0xc009b38000, 0xf1a, 0xf1a, 0x203000, 0x7f6f10227100, 0x20300200000000)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc024bd6e20, 0xc009b38000, 0xf1a, 0xf1a, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
crypto/tls.(*atLeastReader).Read(0xc003906ac0, 0xc009b38000, 0xf1a, 0xf1a, 0x54429a, 0x203001, 0xc000b6c8a0)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x62
bytes.(*Buffer).ReadFrom(0xc006286958, 0x31596e0, 0xc003906ac0, 0x40c417, 0x2d69500, 0x3073620)
	/usr/local/go/src/bytes/buffer.go:204 +0xb6
crypto/tls.(*Conn).readFromUntil(0xc006286700, 0x315d820, 0xc024bd6e20, 0x5, 0xc024bd6e20, 0x40e492)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xee
crypto/tls.(*Conn).readRecordOrCCS(0xc006286700, 0x0, 0x0, 0xc000b6cdd0)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x126
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc006286700, 0xc00a348000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x163
net/http.(*persistConn).Read(0xc00113bc20, 0xc00a348000, 0x1000, 0x1000, 0xc000b6ced8, 0x45f740, 0xc000b6ced8)
	/usr/local/go/src/net/http/transport.go:1758 +0x77
bufio.(*Reader).fill(0xc023f2f740)
	/usr/local/go/src/bufio/bufio.go:100 +0x105
bufio.(*Reader).Peek(0xc023f2f740, 0x1, 0x2, 0x0, 0x0, 0xc01d632300, 0x0)
	/usr/local/go/src/bufio/bufio.go:138 +0x51
net/http.(*persistConn).readLoop(0xc00113bc20)
	/usr/local/go/src/net/http/transport.go:1911 +0x1d8
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1580 +0xb0f

goroutine 382 [select, 1 minutes]:
github.com/docker/docker/vendor/github.com/docker/libnetwork/osl.removeUnusedPaths()
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/osl/namespace_linux.go:105 +0x2dd
created by github.com/docker/docker/vendor/github.com/docker/libnetwork/osl.createBasePath
	/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/osl/namespace_linux.go:90 +0xbd

goroutine 2330503 [IO wait]:
internal/poll.runtime_pollWait(0x7f6e34e372a8, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0094b8918, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0094b8900, 0xc004bb2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc0094b8900, 0xc004bb2000, 0x8000, 0x8000, 0x0, 0x800010601, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc020ad8118, 0xc004bb2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
bufio.(*Reader).Read(0xc01aeb6900, 0xc029f9dc38, 0x9, 0x9, 0xc000dc2000, 0x7f6f48739f80, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x26c
io.ReadAtLeast(0x3159480, 0xc01aeb6900, 0xc029f9dc38, 0x9, 0x9, 0x9, 0xb36af7, 0xc027a6d26c, 0xc003d41de8)
	/usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc029f9dc38, 0x9, 0x9, 0x3159480, 0xc01aeb6900, 0x0, 0xc0239bac00000000, 0xc1bbaf6dfe21, 0x45a58c0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc029f9dc00, 0xc027a6d260, 0xc027a6d260, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0047c8700)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x185
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd25

goroutine 2329596 [IO wait]:
internal/poll.runtime_pollWait(0x7f6f44326850, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0094b8598, 0x72, 0x0, 0x1, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0094b8580, 0xc004133631, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc0094b8580, 0xc004133631, 0x1, 0x1, 0xc01b1be6c0, 0x58da40, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc020ad8110, 0xc004133631, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
net/http.(*connReader).backgroundRead(0xc004133620)
	/usr/local/go/src/net/http/server.go:688 +0x5a
created by net/http.(*connReader).startBackgroundRead
	/usr/local/go/src/net/http/server.go:684 +0xd6

goroutine 2330807 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc01b0ec590, 0xc000000037)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc01b0ec580)
	/usr/local/go/src/sync/cond.go:56 +0x9f
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*progressReader).Read(0xc003d9db90, 0x31bff00, 0xc00011a018, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:132 +0x130
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).handle(0xc003d9dc80, 0xc004133a40, 0xc003d9dc80)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:56 +0x11f
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).Reader
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:47 +0x222

goroutine 2330871 [IO wait]:
internal/poll.runtime_pollWait(0x7f6e34e37038, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc003288a18, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc003288a00, 0xc005c34000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc003288a00, 0xc005c34000, 0x8000, 0x8000, 0xc000000000, 0x800010601, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc024bd62a0, 0xc005c34000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
bufio.(*Reader).Read(0xc01b48a000, 0xc006897998, 0x9, 0x9, 0xc0006e0000, 0x7f6f48739f80, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x26c
io.ReadAtLeast(0x3159480, 0xc01b48a000, 0xc006897998, 0x9, 0x9, 0x9, 0xb36af7, 0xc027a6d28c, 0xc00483ade8)
	/usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc006897998, 0x9, 0x9, 0x3159480, 0xc01b48a000, 0x0, 0xc0239bac00000000, 0xc1bbb0e83117, 0x45a58c0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc006897960, 0xc027a6d280, 0xc027a6d280, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc003f29180)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x185
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd25

goroutine 2330704 [semacquire]:
sync.runtime_Semacquire(0xc003d9da50)
	/usr/local/go/src/runtime/sema.go:56 +0x44
sync.(*WaitGroup).Wait(0xc003d9da48)
	/usr/local/go/src/sync/waitgroup.go:130 +0x66
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc003d9da40, 0xc01107c600, 0xc01ed1c540)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x33
github.com/docker/docker/builder/builder-next.(*Builder).Build(0xc000bc4300, 0x31bfec0, 0xc01b0ec080, 0x31978c0, 0x45cb9e0, 0x31599a0, 0xc01107c4c0, 0x315a240, 0xc01107c4e0, 0x315a240, ...)
	/go/src/github.com/docker/docker/builder/builder-next/builder.go:414 +0xe6d
github.com/docker/docker/api/server/backend/build.(*Backend).Build(0xc000e3a870, 0x31bff80, 0xc003d9d740, 0x31978c0, 0x45cb9e0, 0x31599a0, 0xc01107c4c0, 0x315a240, 0xc01107c4e0, 0x315a240, ...)
	/go/src/github.com/docker/docker/api/server/backend/build/backend.go:64 +0x141
github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild(0xc00092a680, 0x31bff80, 0xc003d9d740, 0x31b0380, 0xc003254620, 0xc003e4c400, 0xc003d9d680, 0x0, 0x0)
	/go/src/github.com/docker/docker/api/server/router/build/build_routes.go:282 +0x731
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x31bff80, 0xc003d9d740, 0x31b0380, 0xc003254620, 0xc003e4c400, 0xc003d9d680, 0x31bff80, 0xc003d9d740)
	/go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x177
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x31bff80, 0xc003d9d710, 0x31b0380, 0xc003254620, 0xc003e4c400, 0xc003d9d680, 0x203004, 0x203004)
	/go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x5fb
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x31bff80, 0xc003d9d710, 0x31b0380, 0xc003254620, 0xc003e4c400, 0xc003d9d680, 0x31bff80, 0xc003d9d710)
	/go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x826
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x31b0380, 0xc003254620, 0xc003e4c300)
	/go/src/github.com/docker/docker/api/server/server.go:141 +0x241
net/http.HandlerFunc.ServeHTTP(0xc000eab580, 0x31b0380, 0xc003254620, 0xc003e4c300)
	/usr/local/go/src/net/http/server.go:2036 +0x46
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00088c240, 0x31b0380, 0xc003254620, 0xc003e84600)
	/go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0xe4
net/http.serverHandler.ServeHTTP(0xc00023a0e0, 0x31b0380, 0xc003254620, 0xc003e84600)
	/usr/local/go/src/net/http/server.go:2831 +0xa6
net/http.(*conn).serve(0xc00820d180, 0x31bfec0, 0xc01b1b2e80)
	/usr/local/go/src/net/http/server.go:1919 +0x877
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2957 +0x386

goroutine 2330857 [chan receive]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.pipe.func1(0x31bfec0, 0xc01ae99540, 0xc004ba0b40)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:172 +0x4e
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.pipe
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:171 +0x131

goroutine 2329598 [semacquire]:
sync.runtime_Semacquire(0xc0041338a0)
	/usr/local/go/src/runtime/sema.go:56 +0x44
sync.(*WaitGroup).Wait(0xc004133898)
	/usr/local/go/src/sync/waitgroup.go:130 +0x66
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc004133890, 0xc00b963780, 0x31bfec0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x33
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status(0xc000836360, 0xc004133860, 0x31e0240, 0xc00b963760, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:362 +0x1a5
github.com/docker/docker/builder/builder-next.(*Builder).Build.func3(0x0, 0x0)
	/go/src/github.com/docker/docker/builder/builder-next/builder.go:398 +0x14b
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc003d9da40, 0xc003d9da70)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 2347995 [select]:
net/http.(*persistConn).writeLoop(0xc02bcfd680)
	/usr/local/go/src/net/http/transport.go:2210 +0x125
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1581 +0xb34

goroutine 2323729 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0068efe60)
	/usr/local/go/src/net/http/transport.go:2210 +0x125
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1581 +0xb34

goroutine 2330504 [select]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00adceaf0, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x124
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc01aeb6a20, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0047c8700)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7d
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xedd

goroutine 2330803 [chan receive]:
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status.func2(0xc001ce9768, 0xc0040a2ae0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:304 +0x7ed
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc004133890, 0xc00b963780)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 2330813 [chan receive]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc0040682c0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x6ec
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x12a

goroutine 2330814 [select]:
github.com/docker/docker/vendor/github.com/moby/buildkit/session.monitorHealth(0x31bfec0, 0xc01afa4d00, 0xc004198a80, 0xc00a47f8c0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/grpc.go:87 +0x1cd
created by github.com/docker/docker/vendor/github.com/moby/buildkit/session.grpcClientConn
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/grpc.go:73 +0x335

goroutine 2323628 [select]:
net/http.(*persistConn).writeLoop(0xc006490fc0)
	/usr/local/go/src/net/http/transport.go:2210 +0x125
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1581 +0xb34

goroutine 2330886 [chan receive]:
github.com/docker/docker/builder/builder-next.(*Builder).Build.func4(0xc000bb3f68, 0x0)
	/go/src/github.com/docker/docker/builder/builder-next/builder.go:402 +0x58
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc004a98e70, 0xc01107d760)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 2323711 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc004a0cc60)
	/usr/local/go/src/net/http/transport.go:2210 +0x125
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1581 +0xb34

goroutine 2331336 [select]:
net/http.(*persistConn).writeLoop(0xc00113bc20)
	/usr/local/go/src/net/http/transport.go:2210 +0x125
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1581 +0xb34

goroutine 2330885 [semacquire]:
sync.runtime_Semacquire(0xc004a98f10)
	/usr/local/go/src/runtime/sema.go:56 +0x44
sync.(*WaitGroup).Wait(0xc004a98f08)
	/usr/local/go/src/sync/waitgroup.go:130 +0x66
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc004a98f00, 0xc01107d840, 0x31bfec0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x33
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status(0xc000836360, 0xc004a98ed0, 0x31e0240, 0xc01107d800, 0x0, 0xc02a71f788)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:362 +0x1a5
github.com/docker/docker/builder/builder-next.(*Builder).Build.func3(0x0, 0x0)
	/go/src/github.com/docker/docker/builder/builder-next/builder.go:398 +0x14b
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc004a98e70, 0xc004a98ea0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 2330812 [select]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc01afa4a80)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc4
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x16f

goroutine 2347769 [sleep]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
time.Sleep(0x2540be400)
	/usr/local/go/src/runtime/time.go:105 +0x15d
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*Job).Discard.func1(0xc00924d980)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/jobs.go:532 +0x36
created by github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*Job).Discard
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/jobs.go:530 +0x3b3

goroutine 2330806 [select]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).Reader.func1(0x31bff80, 0xc004133a40, 0xc003d9dc80, 0xc00b9638a0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:37 +0xc3
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).Reader
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:36 +0x1c4

goroutine 2329600 [chan receive]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.pipe.func1(0x31bfec0, 0xc01b0ec500, 0xc003d9db90)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:172 +0x4e
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.pipe
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:171 +0x131

goroutine 2330859 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc01ae44750, 0xc000000038)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc01ae44740)
	/usr/local/go/src/sync/cond.go:56 +0x9f
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*progressReader).Read(0xc004b7ca80, 0x31bff00, 0xc00011a018, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:132 +0x130
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).handle(0xc004b7cb70, 0xc004ba0bd0, 0xc004b7cb70)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:56 +0x11f
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).Reader
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:47 +0x222

goroutine 2347928 [select]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*progressReader).Read.func1(0xc00d288d20, 0x31bfec0, 0xc01b1b36c0, 0xc0041339b0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:106 +0xb9
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*progressReader).Read
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:105 +0xf7

goroutine 2347880 [select]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*progressReader).Read.func1(0xc00d191020, 0x31bff00, 0xc00011a018, 0xc004b7ca80)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:106 +0xb9
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*progressReader).Read
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:105 +0xf7

goroutine 2330888 [chan receive]:
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status.func2(0xc00176d768, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:304 +0x7ed
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc004a98f00, 0xc01107d840)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 2330883 [IO wait]:
internal/poll.runtime_pollWait(0x7f6f4415e130, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc004d18918, 0x72, 0x0, 0x1, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc004d18900, 0xc003d79091, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc004d18900, 0xc003d79091, 0x1, 0x1, 0xc00849f8b0, 0xc01c009100, 0xc02a71c7c0)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc02758c020, 0xc003d79091, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
net/http.(*connReader).backgroundRead(0xc003d79080)
	/usr/local/go/src/net/http/server.go:688 +0x5a
created by net/http.(*connReader).startBackgroundRead
	/usr/local/go/src/net/http/server.go:684 +0xd6

goroutine 2330869 [chan receive]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc00217f080)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x6ec
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x12a

goroutine 2331358 [IO wait]:
internal/poll.runtime_pollWait(0x7f6f443266b0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc003091818, 0x72, 0xf00, 0xf1a, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc003091800, 0xc0097f9000, 0xf1a, 0xf1a, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc003091800, 0xc0097f9000, 0xf1a, 0xf1a, 0x203002, 0x0, 0xc00ad4d7e8)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc0181ad8a0, 0xc0097f9000, 0xf1a, 0xf1a, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
crypto/tls.(*atLeastReader).Read(0xc00a8b3040, 0xc0097f9000, 0xf1a, 0xf1a, 0x54429a, 0x203002, 0xc00ad4d8a0)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x62
bytes.(*Buffer).ReadFrom(0xc009994258, 0x31596e0, 0xc00a8b3040, 0x40c417, 0x2d69500, 0x3073620)
	/usr/local/go/src/bytes/buffer.go:204 +0xb6
crypto/tls.(*Conn).readFromUntil(0xc009994000, 0x315d820, 0xc0181ad8a0, 0x5, 0xc0181ad8a0, 0x40e492)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xee
crypto/tls.(*Conn).readRecordOrCCS(0xc009994000, 0x0, 0x0, 0xc00ad4ddd0)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x126
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc009994000, 0xc00a3c6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x163
net/http.(*persistConn).Read(0xc0162bd9e0, 0xc00a3c6000, 0x1000, 0x1000, 0xc00ad4ded8, 0x45f740, 0xc00ad4ded8)
	/usr/local/go/src/net/http/transport.go:1758 +0x77
bufio.(*Reader).fill(0xc02409f260)
	/usr/local/go/src/bufio/bufio.go:100 +0x105
bufio.(*Reader).Peek(0xc02409f260, 0x1, 0x2, 0x0, 0x0, 0xc01d6f7000, 0x0)
	/usr/local/go/src/bufio/bufio.go:138 +0x51
net/http.(*persistConn).readLoop(0xc0162bd9e0)
	/usr/local/go/src/net/http/transport.go:1911 +0x1d8
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1580 +0xb0f

goroutine 2323710 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f6f44341e60, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc00dcff118, 0x72, 0xf00, 0xf1a, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00dcff100, 0xc00200c000, 0xf1a, 0xf1a, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc00dcff100, 0xc00200c000, 0xf1a, 0xf1a, 0x20300a, 0x2c0010000000000, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc01495c538, 0xc00200c000, 0xf1a, 0xf1a, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
crypto/tls.(*atLeastReader).Read(0xc02ae56180, 0xc00200c000, 0xf1a, 0xf1a, 0x45ac618, 0x7f6ee80e26e0, 0xc00ad4a8a0)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x62
bytes.(*Buffer).ReadFrom(0xc0130465d8, 0x31596e0, 0xc02ae56180, 0x40c417, 0x2d69500, 0x3073620)
	/usr/local/go/src/bytes/buffer.go:204 +0xb6
crypto/tls.(*Conn).readFromUntil(0xc013046380, 0x315d820, 0xc01495c538, 0x5, 0xc01495c538, 0x40e492)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xee
crypto/tls.(*Conn).readRecordOrCCS(0xc013046380, 0x0, 0x0, 0xc00ad4add0)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x126
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc013046380, 0xc0020e4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x163
net/http.(*persistConn).Read(0xc004a0cc60, 0xc0020e4000, 0x1000, 0x1000, 0xc00ad4aed8, 0x45f740, 0xc00ad4aed8)
	/usr/local/go/src/net/http/transport.go:1758 +0x77
bufio.(*Reader).fill(0xc0274c7e60)
	/usr/local/go/src/bufio/bufio.go:100 +0x105
bufio.(*Reader).Peek(0xc0274c7e60, 0x1, 0x2, 0x0, 0x0, 0xc029376a00, 0x0)
	/usr/local/go/src/bufio/bufio.go:138 +0x51
net/http.(*persistConn).readLoop(0xc004a0cc60)
	/usr/local/go/src/net/http/transport.go:1911 +0x1d8
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1580 +0xb0f

goroutine 2330858 [select]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).Reader.func1(0x31bff80, 0xc004ba0bd0, 0xc004b7cb70, 0xc01195bce0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:37 +0xc3
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).Reader
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:36 +0x1c4

goroutine 2347770 [select]:
net/http.(*persistConn).roundTrip(0xc02bcfd680, 0xc00881ccc0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/transport.go:2431 +0x772
net/http.(*Transport).roundTrip(0xc002c5f7c0, 0xc002f62d00, 0xc02ceacac8, 0xc00a0a1fe8, 0x0)
	/usr/local/go/src/net/http/transport.go:535 +0x9b8
net/http.(*Transport).RoundTrip(0xc002c5f7c0, 0xc002f62d00, 0xc002f62d00, 0x0, 0xc0006e0000)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x37
github.com/docker/docker/vendor/github.com/docker/distribution/registry/client/transport.(*transport).RoundTrip(0xc00a0a1fc0, 0xc002f62c00, 0xc00a0a1fc0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/docker/distribution/registry/client/transport/transport.go:62 +0xfd
net/http.send(0xc002f62c00, 0x315ad40, 0xc00a0a1fc0, 0x0, 0x0, 0x0, 0xc001c9ba18, 0x203002, 0x1, 0x0)
	/usr/local/go/src/net/http/client.go:250 +0x445
net/http.(*Client).send(0xc0022d31a0, 0xc002f62c00, 0x0, 0x0, 0x0, 0xc001c9ba18, 0x0, 0x1, 0x30d82c0)
	/usr/local/go/src/net/http/client.go:174 +0xfc
net/http.(*Client).do(0xc0022d31a0, 0xc002f62c00, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:641 +0x3d0
net/http.(*Client).Do(...)
	/usr/local/go/src/net/http/client.go:509
net/http.(*Client).Post(0xc0022d31a0, 0xc00da835c0, 0x3c, 0x0, 0x0, 0x0, 0x0, 0x3c, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:769 +0x197
github.com/docker/docker/vendor/github.com/docker/distribution/registry/client.(*blobs).Create(0xc009d2bf40, 0x31bfec0, 0xc00a0a1b40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/docker/distribution/registry/client/repository.go:739 +0x19a
github.com/docker/docker/vendor/github.com/docker/distribution/registry/client.(*blobs).Put(0xc009d2bf40, 0x31bfec0, 0xc00a0a1b40, 0x20a73fc, 0x2e, 0xc036082a80, 0x4859, 0x4859, 0x0, 0x0, ...)
	/go/src/github.com/docker/docker/vendor/github.com/docker/distribution/registry/client/repository.go:674 +0x8f
github.com/docker/docker/vendor/github.com/docker/distribution/manifest/schema2.(*builder).Build(0xc01984ad70, 0x31bfec0, 0xc00a0a1b40, 0x0, 0x0, 0x2ee3480, 0xc0084c4201)
	/go/src/github.com/docker/docker/vendor/github.com/docker/distribution/manifest/schema2/builder.go:65 +0x2ac
github.com/docker/docker/distribution.manifestFromBuilder(0x31bfec0, 0xc00a0a1b40, 0x31a3a80, 0xc01984ad70, 0xc002f62700, 0xe, 0x10, 0xc00dcfa0c0, 0x0, 0x0, ...)
	/go/src/github.com/docker/docker/distribution/push_v2.go:260 +0xf4
github.com/docker/docker/distribution.(*v2Pusher).pushV2Tag(0xc012988500, 0x31bfec0, 0xc00a0a1b40, 0x31b1080, 0xc001f21f60, 0xc01d75a2d0, 0x47, 0x0, 0x0)
	/go/src/github.com/docker/docker/distribution/push_v2.go:173 +0xd2b
github.com/docker/docker/distribution.(*v2Pusher).pushV2Repository(0xc012988500, 0x31bfec0, 0xc00a0a1b40, 0x0, 0xc012988380)
	/go/src/github.com/docker/docker/distribution/push_v2.go:91 +0x1f2
github.com/docker/docker/distribution.(*v2Pusher).Push(0xc012988500, 0x31bfec0, 0xc00a0a1b40, 0xc012988380, 0x2)
	/go/src/github.com/docker/docker/distribution/push_v2.go:72 +0x264
github.com/docker/docker/distribution.Push(0x31bfec0, 0xc00a0a1b40, 0x31969c0, 0xc001f21f60, 0xc00305d2c0, 0xc00dcfa0c0, 0xc021be4cc0)
	/go/src/github.com/docker/docker/distribution/push.go:110 +0x619
github.com/docker/docker/daemon/images.(*ImageService).PushImage(0xc000b66000, 0x31bfec0, 0xc00a0a1b40, 0xc02a62bf93, 0x22, 0xc02a62bfbf, 0x1a, 0xc0169df050, 0xc0008931f0, 0x315a160, ...)
	/go/src/github.com/docker/docker/daemon/images/image_push.go:61 +0x3c7
github.com/docker/docker/api/server/router/image.(*imageRouter).postImagesPush(0xc000e3b320, 0x31bff80, 0xc0169df020, 0x31b0380, 0xc002f07500, 0xc002f62100, 0xc0169def60, 0x0, 0x0)
	/go/src/github.com/docker/docker/api/server/router/image/image_routes.go:135 +0x742
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x31bff80, 0xc0169df020, 0x31b0380, 0xc002f07500, 0xc002f62100, 0xc0169def60, 0x31bff80, 0xc0169df020)
	/go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x177
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x31bff80, 0xc0169deff0, 0x31b0380, 0xc002f07500, 0xc002f62100, 0xc0169def60, 0x203000, 0x203000)
	/go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x5fb
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x31bff80, 0xc0169deff0, 0x31b0380, 0xc002f07500, 0xc002f62100, 0xc0169def60, 0x31bff80, 0xc0169deff0)
	/go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x826
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x31b0380, 0xc002f07500, 0xc002f62000)
	/go/src/github.com/docker/docker/api/server/server.go:141 +0x241
net/http.HandlerFunc.ServeHTTP(0xc000e614a0, 0x31b0380, 0xc002f07500, 0xc002f62000)
	/usr/local/go/src/net/http/server.go:2036 +0x46
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00088c240, 0x31b0380, 0xc002f07500, 0xc00c609e00)
	/go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0xe4
net/http.serverHandler.ServeHTTP(0xc00023a0e0, 0x31b0380, 0xc002f07500, 0xc00c609e00)
	/usr/local/go/src/net/http/server.go:2831 +0xa6
net/http.(*conn).serve(0xc00305d220, 0x31bfec0, 0xc00a0a1980)
	/usr/local/go/src/net/http/server.go:1919 +0x877
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2957 +0x386

goroutine 2329601 [chan receive]:
github.com/docker/docker/vendor/github.com/moby/buildkit/session.(*Manager).handleConn(0xc00000f500, 0x31bfec0, 0xc01ac8e480, 0x31ea680, 0xc024bd62a0, 0xc004a98840, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/manager.go:144 +0x4c4
github.com/docker/docker/vendor/github.com/moby/buildkit/session.(*Manager).HandleHTTPRequest(0xc00000f500, 0x31bff80, 0xc004a989c0, 0x31b0380, 0xc003254700, 0xc003e4c900, 0xc000dcc700, 0x415434)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/manager.go:91 +0x7cf
github.com/docker/docker/api/server/router/session.(*sessionRouter).startSession(0xc000e3b8f0, 0x31bff80, 0xc004a989c0, 0x31b0380, 0xc003254700, 0xc003e4c900, 0xc004a98900, 0xc000dcc701, 0xc009cec160)
	/go/src/github.com/docker/docker/api/server/router/session/session_routes.go:11 +0x6b
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x31bff80, 0xc004a989c0, 0x31b0380, 0xc003254700, 0xc003e4c900, 0xc004a98900, 0x31bff80, 0xc004a989c0)
	/go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x177
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x31bff80, 0xc004a98990, 0x31b0380, 0xc003254700, 0xc003e4c900, 0xc004a98900, 0x203004, 0x203004)
	/go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x5fb
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x31bff80, 0xc004a98990, 0x31b0380, 0xc003254700, 0xc003e4c900, 0xc004a98900, 0x31bff80, 0xc004a98990)
	/go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x826
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x31b0380, 0xc003254700, 0xc003e4c800)
	/go/src/github.com/docker/docker/api/server/server.go:141 +0x241
net/http.HandlerFunc.ServeHTTP(0xc000eaba60, 0x31b0380, 0xc003254700, 0xc003e4c800)
	/usr/local/go/src/net/http/server.go:2036 +0x46
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00088c240, 0x31b0380, 0xc003254700, 0xc003e4c500)
	/go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0xe4
net/http.serverHandler.ServeHTTP(0xc00023a0e0, 0x31b0380, 0xc003254700, 0xc003e4c500)
	/usr/local/go/src/net/http/server.go:2831 +0xa6
net/http.(*conn).serve(0xc00dc8f7c0, 0x31bfec0, 0xc01b0ed940)
	/usr/local/go/src/net/http/server.go:1919 +0x877
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2957 +0x386

goroutine 2347977 [select]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*progressReader).Read.func1(0xc0098f8de0, 0x31bfec0, 0xc01ad69400, 0xc004ba0b40)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:106 +0xb9
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*progressReader).Read
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:105 +0xf7

goroutine 2347994 [IO wait]:
internal/poll.runtime_pollWait(0x7f6f4415db80, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0030f5918, 0x72, 0x1400, 0x1488, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0030f5900, 0xc002d1f500, 0x1488, 0x1488, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc0030f5900, 0xc002d1f500, 0x1488, 0x1488, 0x203004, 0x0, 0x1483)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc001c9ba80, 0xc002d1f500, 0x1488, 0x1488, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
crypto/tls.(*atLeastReader).Read(0xc011123ae0, 0xc002d1f500, 0x1488, 0x1488, 0x4f, 0x3159920, 0xc02fdea8a0)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x62
bytes.(*Buffer).ReadFrom(0xc00308de58, 0x31596e0, 0xc011123ae0, 0x40c417, 0x2d69500, 0x3073620)
	/usr/local/go/src/bytes/buffer.go:204 +0xb6
crypto/tls.(*Conn).readFromUntil(0xc00308dc00, 0x315d820, 0xc001c9ba80, 0x5, 0xc001c9ba80, 0x58)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xee
crypto/tls.(*Conn).readRecordOrCCS(0xc00308dc00, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x126
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc00308dc00, 0xc003aab000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x163
net/http.(*persistConn).Read(0xc02bcfd680, 0xc003aab000, 0x1000, 0x1000, 0x406aff, 0x60, 0x0)
	/usr/local/go/src/net/http/transport.go:1758 +0x77
bufio.(*Reader).fill(0xc01dd8b4a0)
	/usr/local/go/src/bufio/bufio.go:100 +0x105
bufio.(*Reader).Peek(0xc01dd8b4a0, 0x1, 0xc00cf591a0, 0xc00176c498, 0x5ad48f, 0x2e262a0, 0x14)
	/usr/local/go/src/bufio/bufio.go:138 +0x51
net/http.(*persistConn).readLoop(0xc02bcfd680)
	/usr/local/go/src/net/http/transport.go:1911 +0x1d8
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1580 +0xb0f

goroutine 2330870 [select]:
github.com/docker/docker/vendor/github.com/moby/buildkit/session.monitorHealth(0x31bfec0, 0xc01ac8e480, 0xc00095fc00, 0xc031a13de0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/grpc.go:87 +0x1cd
created by github.com/docker/docker/vendor/github.com/moby/buildkit/session.grpcClientConn
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/grpc.go:73 +0x335

goroutine 2347772 [IO wait]:
internal/poll.runtime_pollWait(0x7f6e34e37448, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc02f138d98, 0x72, 0x0, 0x1, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc02f138d80, 0xc0169deb81, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc02f138d80, 0xc0169deb81, 0x1, 0x1, 0xc00a0a19d8, 0xc000bb2f68, 0x47d98e)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc001c9b748, 0xc0169deb81, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
net/http.(*connReader).backgroundRead(0xc0169deb70)
	/usr/local/go/src/net/http/server.go:688 +0x5a
created by net/http.(*connReader).startBackgroundRead
	/usr/local/go/src/net/http/server.go:684 +0xd6

goroutine 2323728 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f6e3769d618, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc01719b418, 0x72, 0xf00, 0xf1a, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc01719b400, 0xc00200d000, 0xf1a, 0xf1a, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc01719b400, 0xc00200d000, 0xf1a, 0xf1a, 0x203001, 0x0, 0xc00ad4b7e8)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc00b6a6520, 0xc00200d000, 0xf1a, 0xf1a, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
crypto/tls.(*atLeastReader).Read(0xc00408c680, 0xc00200d000, 0xf1a, 0xf1a, 0x54429a, 0x203002, 0xc00ad4b8a0)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x62
bytes.(*Buffer).ReadFrom(0xc0033573d8, 0x31596e0, 0xc00408c680, 0x40c417, 0x2d69500, 0x3073620)
	/usr/local/go/src/bytes/buffer.go:204 +0xb6
crypto/tls.(*Conn).readFromUntil(0xc003357180, 0x315d820, 0xc00b6a6520, 0x5, 0xc00b6a6520, 0x40e492)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xee
crypto/tls.(*Conn).readRecordOrCCS(0xc003357180, 0x0, 0x0, 0xc00ad4bdd0)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x126
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc003357180, 0xc001e9f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x163
net/http.(*persistConn).Read(0xc0068efe60, 0xc001e9f000, 0x1000, 0x1000, 0xc00ad4bed8, 0x45f740, 0xc00ad4bed8)
	/usr/local/go/src/net/http/transport.go:1758 +0x77
bufio.(*Reader).fill(0xc01373c000)
	/usr/local/go/src/bufio/bufio.go:100 +0x105
bufio.(*Reader).Peek(0xc01373c000, 0x1, 0x2, 0x0, 0x0, 0xc029343a00, 0x0)
	/usr/local/go/src/bufio/bufio.go:138 +0x51
net/http.(*persistConn).readLoop(0xc0068efe60)
	/usr/local/go/src/net/http/transport.go:1911 +0x1d8
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1580 +0xb0f

goroutine 2347952 [select]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*progressReader).Read.func1(0xc01e2e9320, 0x31bff00, 0xc00011a018, 0xc003d9db90)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:106 +0xb9
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*progressReader).Read
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:105 +0xf7

goroutine 2330802 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc01b1b3990, 0xc000000033)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc01b1b3980)
	/usr/local/go/src/sync/cond.go:56 +0x9f
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*progressReader).Read(0xc0041339b0, 0x31bfec0, 0xc01b1b36c0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:132 +0x130
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*Job).Status(0xc003288880, 0x31bfec0, 0xc01b1b36c0, 0xc01afd6ba0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/progress.go:25 +0xd4b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.(*Solver).Status(0xc000e42a20, 0x31bfec0, 0xc01b1b36c0, 0xc02e38b061, 0x19, 0xc01afd6ba0, 0xc01b1b3398, 0xc0036cbf68)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:306 +0xc0
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status.func1(0xc0036cbf68, 0x7f6f442e7901)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:299 +0x60
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc004133890, 0xc0041338c0)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 2330887 [sync.Cond.Wait]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc01ae99650, 0xc000000033)
	/usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc01ae99640)
	/usr/local/go/src/sync/cond.go:56 +0x9f
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*progressReader).Read(0xc004ba0b40, 0x31bfec0, 0xc01ad69400, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:132 +0x130
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*Job).Status(0xc0094b9080, 0x31bfec0, 0xc01ad69400, 0xc01b3c43c0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/progress.go:25 +0xd4b
github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.(*Solver).Status(0xc000e42a20, 0x31bfec0, 0xc01ad69400, 0xc016a90b01, 0x19, 0xc01b3c43c0, 0x3115bc8, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:306 +0xc0
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status.func1(0xc00176b768, 0xc00176b710)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:299 +0x60
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc004a98f00, 0xc004a98f30)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 2330824 [sleep]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
time.Sleep(0xdf8475800)
	/usr/local/go/src/runtime/time.go:105 +0x15d
github.com/docker/docker/vendor/github.com/moby/buildkit/util/throttle.throttle.func1.1(0xc000e38318, 0xc000e38310, 0xc000e38311, 0x0, 0xdf8475800, 0xc0008fc2e0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/throttle/throttle.go:52 +0x12e
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/throttle.throttle.func1
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/throttle/throttle.go:31 +0x13f

goroutine 2330868 [select]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc01ac8e040)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc4
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x16f

goroutine 2330822 [chan receive]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.pipe.func1(0x31bfec0, 0xc01ae44280, 0xc004b7ca80)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:172 +0x4e
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.pipe
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:171 +0x131

goroutine 2330525 [sleep]:
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:310
time.Sleep(0x2540be400)
	/usr/local/go/src/runtime/time.go:105 +0x15d
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*Job).Discard.func1(0xc016fd9b80)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/jobs.go:532 +0x36
created by github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*Job).Discard
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/jobs.go:530 +0x3b3

goroutine 2330810 [chan receive]:
github.com/docker/docker/vendor/github.com/moby/buildkit/session.(*Manager).handleConn(0xc00000f500, 0x31bfec0, 0xc01afa4d00, 0x31ea680, 0xc020ad8118, 0xc004133b60, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/manager.go:144 +0x4c4
github.com/docker/docker/vendor/github.com/moby/buildkit/session.(*Manager).HandleHTTPRequest(0xc00000f500, 0x31bff80, 0xc004133ce0, 0x31b0380, 0xc00210e620, 0xc003e84b00, 0xc000df2380, 0x415434)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/manager.go:91 +0x7cf
github.com/docker/docker/api/server/router/session.(*sessionRouter).startSession(0xc000e3b8f0, 0x31bff80, 0xc004133ce0, 0x31b0380, 0xc00210e620, 0xc003e84b00, 0xc004133c20, 0xc000df2301, 0xc00ab7f080)
	/go/src/github.com/docker/docker/api/server/router/session/session_routes.go:11 +0x6b
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x31bff80, 0xc004133ce0, 0x31b0380, 0xc00210e620, 0xc003e84b00, 0xc004133c20, 0x31bff80, 0xc004133ce0)
	/go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x177
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x31bff80, 0xc004133cb0, 0x31b0380, 0xc00210e620, 0xc003e84b00, 0xc004133c20, 0x203002, 0x203002)
	/go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x5fb
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x31bff80, 0xc004133cb0, 0x31b0380, 0xc00210e620, 0xc003e84b00, 0xc004133c20, 0x31bff80, 0xc004133cb0)
	/go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x826
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x31b0380, 0xc00210e620, 0xc003e84a00)
	/go/src/github.com/docker/docker/api/server/server.go:141 +0x241
net/http.HandlerFunc.ServeHTTP(0xc000eaba60, 0x31b0380, 0xc00210e620, 0xc003e84a00)
	/usr/local/go/src/net/http/server.go:2036 +0x46
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00088c240, 0x31b0380, 0xc00210e620, 0xc003e84700)
	/go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0xe4
net/http.serverHandler.ServeHTTP(0xc00023a0e0, 0x31b0380, 0xc00210e620, 0xc003e84700)
	/usr/local/go/src/net/http/server.go:2831 +0xa6
net/http.(*conn).serve(0xc00820d220, 0x31bfec0, 0xc01b1b3a00)
	/usr/local/go/src/net/http/server.go:1919 +0x877
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2957 +0x386

goroutine 2329599 [chan receive]:
github.com/docker/docker/builder/builder-next.(*Builder).Build.func4(0xc02a71df68, 0xc018be79e0)
	/go/src/github.com/docker/docker/builder/builder-next/builder.go:402 +0x58
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc003d9da40, 0xc01107c600)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 2329597 [runnable]:
github.com/docker/docker/vendor/go.etcd.io/bbolt.cloneBytes(...)
	/go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/bucket.go:774
github.com/docker/docker/vendor/go.etcd.io/bbolt.(*Bucket).openBucket(0xc0250bba80, 0x7f6e159be4b9, 0x296, 0x296, 0x7f6e159be4a0)
	/go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/bucket.go:133 +0x259
github.com/docker/docker/vendor/go.etcd.io/bbolt.(*Bucket).Bucket(0xc0250bba80, 0xc003e44040, 0x19, 0x20, 0x19)
	/go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/bucket.go:113 +0x149
github.com/docker/docker/vendor/github.com/moby/buildkit/solver/bboltcachestorage.(*Store).WalkBacklinks.func1(0xc003c22380, 0x416100, 0xc003c22380)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/bboltcachestorage/storage.go:400 +0x162
github.com/docker/docker/vendor/go.etcd.io/bbolt.(*DB).View(0xc000532e00, 0xc003e441f8, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:725 +0xaa
github.com/docker/docker/vendor/github.com/moby/buildkit/solver/bboltcachestorage.(*Store).WalkBacklinks(0xc000010338, 0xc0046ddc20, 0x19, 0xc0201f9050, 0xc003c08658, 0x411199)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/bboltcachestorage/storage.go:391 +0xd9
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.addBacklinks(0x7f6f443409a0, 0xc01107c700, 0x31920c0, 0xc020552c60, 0xc00045e0a0, 0xc0046ddc20, 0x19, 0xc013c4b0e0, 0xc01a5de320, 0xc00c9d9820, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:29 +0x159
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc007522500, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0xc037294490, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:164 +0x446
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc007522af0, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0xc00066a000, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc029715e00, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0xc01a5de320, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc0121a0dc0, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0xc01a5de320, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc006a270e0, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0xc01a5de320, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc015294f00, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0x5, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc00707c8c0, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0x5, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc00707caf0, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0x5, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc00707d1d0, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0x5, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc00707d450, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0x5, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc00707d900, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0x5, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc00707def0, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0x5, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc0130002d0, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0x5, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc0130005a0, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0x5, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc013000a00, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0xc00066a000, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc013000eb0, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0xc01a5de320, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*exporter).ExportTo(0xc0074dd950, 0x31bff80, 0xc013c4b140, 0x7f6f443409a0, 0xc01107c700, 0xc01a5de360, 0x2, 0x315bda0, 0xc01a5de320, 0x0, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/exporter.go:135 +0xcab
github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.inlineCache(0x31bfec0, 0xc01b0ec340, 0x31c0a00, 0xc01107c700, 0x31c7b00, 0xc01a7184e0, 0x315bda0, 0xc01a5de320, 0x0, 0x0, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:287 +0x350
github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0xc000e42a20, 0x31bfec0, 0xc01b0ec340, 0xc02e38b061, 0x19, 0xc004a5c1e4, 0x19, 0x0, 0x0, 0x2045d85, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:176 +0x15f2
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Solve(0xc000836360, 0x31bfec0, 0xc01b0ec340, 0xc01cc866c0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:275 +0x4fc
github.com/docker/docker/builder/builder-next.(*Builder).Build.func2(0xc0037ca768, 0xc00095f528)
	/go/src/github.com/docker/docker/builder/builder-next/builder.go:376 +0x76
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc003d9da40, 0xc0074dde00)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 2330872 [select]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc011acfb30, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x124
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc01b48a0c0, 0x0, 0x0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc003f29180)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7d
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xedd

goroutine 2330128 [semacquire]:
sync.runtime_Semacquire(0xc004a98e80)
	/usr/local/go/src/runtime/sema.go:56 +0x44
sync.(*WaitGroup).Wait(0xc004a98e78)
	/usr/local/go/src/sync/waitgroup.go:130 +0x66
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc004a98e70, 0xc01107d760, 0xc01ed1de60)
	/go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x33
github.com/docker/docker/builder/builder-next.(*Builder).Build(0xc000bc4300, 0x31bfec0, 0xc01ad68b40, 0x31978c0, 0x45cb9e0, 0x31599a0, 0xc01107d600, 0x315a240, 0xc01107d620, 0x315a240, ...)
	/go/src/github.com/docker/docker/builder/builder-next/builder.go:414 +0xe6d
github.com/docker/docker/api/server/backend/build.(*Backend).Build(0xc000e3a870, 0x31bff80, 0xc004a98b70, 0x31978c0, 0x45cb9e0, 0x31599a0, 0xc01107d600, 0x315a240, 0xc01107d620, 0x315a240, ...)
	/go/src/github.com/docker/docker/api/server/backend/build/backend.go:64 +0x141
github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild(0xc00092a680, 0x31bff80, 0xc004a98b70, 0x31b0380, 0xc0032547e0, 0xc003e4cc00, 0xc004a98ab0, 0x0, 0x0)
	/go/src/github.com/docker/docker/api/server/router/build/build_routes.go:282 +0x731
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x31bff80, 0xc004a98b70, 0x31b0380, 0xc0032547e0, 0xc003e4cc00, 0xc004a98ab0, 0x31bff80, 0xc004a98b70)
	/go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x177
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x31bff80, 0xc004a98b40, 0x31b0380, 0xc0032547e0, 0xc003e4cc00, 0xc004a98ab0, 0x203004, 0x203004)
	/go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x5fb
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x31bff80, 0xc004a98b40, 0x31b0380, 0xc0032547e0, 0xc003e4cc00, 0xc004a98ab0, 0x31bff80, 0xc004a98b40)
	/go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x826
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x31b0380, 0xc0032547e0, 0xc003e4cb00)
	/go/src/github.com/docker/docker/api/server/server.go:141 +0x241
net/http.HandlerFunc.ServeHTTP(0xc000eab580, 0x31b0380, 0xc0032547e0, 0xc003e4cb00)
	/usr/local/go/src/net/http/server.go:2036 +0x46
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00088c240, 0x31b0380, 0xc0032547e0, 0xc003eb2a00)
	/go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0xe4
net/http.serverHandler.ServeHTTP(0xc00023a0e0, 0x31b0380, 0xc0032547e0, 0xc003eb2a00)
	/usr/local/go/src/net/http/server.go:2831 +0xa6
net/http.(*conn).serve(0xc002c98500, 0x31bfec0, 0xc01c28f540)
	/usr/local/go/src/net/http/server.go:1919 +0x877
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2957 +0x386

goroutine 2331359 [select]:
net/http.(*persistConn).writeLoop(0xc0162bd9e0)
	/usr/local/go/src/net/http/transport.go:2210 +0x125
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1581 +0xb34

goroutine 2323627 [IO wait]:
internal/poll.runtime_pollWait(0x7f6f444f6438, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc01396da98, 0x72, 0x1400, 0x1488, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc01396da80, 0xc00214a000, 0x1488, 0x1488, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc01396da80, 0xc00214a000, 0x1488, 0x1488, 0x203005, 0x0, 0xc0010177e8)
	/usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc001c842b0, 0xc00214a000, 0x1488, 0x1488, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x6a
crypto/tls.(*atLeastReader).Read(0xc0179610e0, 0xc00214a000, 0x1488, 0x1488, 0x54429a, 0x203002, 0xc0010178a0)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x62
bytes.(*Buffer).ReadFrom(0xc0016a2258, 0x31596e0, 0xc0179610e0, 0x40c417, 0x2d69500, 0x3073620)
	/usr/local/go/src/bytes/buffer.go:204 +0xb6
crypto/tls.(*Conn).readFromUntil(0xc0016a2000, 0x315d820, 0xc001c842b0, 0x5, 0xc001c842b0, 0x18)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xee
crypto/tls.(*Conn).readRecordOrCCS(0xc0016a2000, 0x0, 0x0, 0xc001017dd0)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x126
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc0016a2000, 0xc001f4b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x163
net/http.(*persistConn).Read(0xc006490fc0, 0xc001f4b000, 0x1000, 0x1000, 0xc001017ed8, 0x45f740, 0xc001017ed8)
	/usr/local/go/src/net/http/transport.go:1758 +0x77
bufio.(*Reader).fill(0xc00e1dbf80)
	/usr/local/go/src/bufio/bufio.go:100 +0x105
bufio.(*Reader).Peek(0xc00e1dbf80, 0x1, 0x2, 0x0, 0x0, 0xc01d795300, 0x0)
	/usr/local/go/src/bufio/bufio.go:138 +0x51
net/http.(*persistConn).readLoop(0xc006490fc0)
	/usr/local/go/src/net/http/transport.go:1911 +0x1d8
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1580 +0xb0f

goroutine 2347773 [chan receive]:
github.com/docker/docker/distribution/utils.WriteDistributionProgress(0xc0233a8180, 0x315a160, 0xc02d2ac0a0, 0xc00dcfa0c0)
	/go/src/github.com/docker/docker/distribution/utils/progress.go:20 +0xdc
github.com/docker/docker/daemon/images.(*ImageService).PushImage.func1(0xc0233a8180, 0x315a160, 0xc02d2ac0a0, 0xc00dcfa0c0, 0xc021be4cc0)
	/go/src/github.com/docker/docker/daemon/images/image_push.go:40 +0x4b
created by github.com/docker/docker/daemon/images.(*ImageService).PushImage
	/go/src/github.com/docker/docker/daemon/images/image_push.go:39 +0x180

goroutine 2330805 [chan receive]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.pipe.func1(0x31bfec0, 0xc01b1b38c0, 0xc0041339b0)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:172 +0x4e
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.pipe
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/progress.go:171 +0x131

Describe the results you received: Docker pod has failed Liveness/Readiness after that probes which are:

livenessProbe:
  httpGet:
    path: /_ping
    port: 2375
readinessProbe:
  httpGet:
    path: /_ping
    port: 2375

and was killed by K8s

Describe the results you expected: Docker operates normally

Additional information you deem important (e.g. issue happens only occasionally): Issue happens very infrequently

Output of docker version:

Client:
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        370c289
 Built:             Fri Apr  9 22:42:10 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:46:14 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 
Server: 
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 189
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.12.1-1.el7.elrepo.x86_64
 Operating System: Alpine Linux v3.13 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 125.7GiB
 Name: production-app-gitlab-runners-docker-2
 ID: WDVE:XF3M:UUBU:YNOD:PUNM:EBCV:L6ED:FKXW:HGO4:IGTX:RH62:BUGB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://dockerhub.xxx/
 Live Restore Enabled: false
 Product License: Community Engine
 Default Address Pools:
   Base: 10.10.0.0/16, Size: 24

WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/

Additional environment details (AWS, VirtualBox, physical, etc.):

Using docker:20.10.6-dind from hub.docker.com

kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:28:42Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:19:55Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
Linux worker-2.prod.xxx.xxx 5.12.1-1.el7.elrepo.x86_64 #1 SMP Fri Apr 30 15:49:43 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 16 (1 by maintainers)

Most upvoted comments

FYI - I am getting email alerts for issues like this one that I have not said I wanted to watch. Am I missing some global setting that has been selected ?

Any news on that? Is it backported to the required BuildKit version and we just wait for the new Docker release? Are there any builds docker-dind with this fix I could take for testing?

I see https://github.com/moby/buildkit/pull/2051 added synchronisation to fix https://github.com/moby/buildkit/issues/2041 (looks like that may be the same issue?). That fix is available in the BuildKit master branch, but not yet back ported to the v0.8 version (which is what’s used by Docker 20.10)

edit: https://github.com/moby/buildkit/pull/2067 is also linked to that issue