traefik: fatal error: concurrent map iteration and map write

Do you want to request a feature or report a bug?

Bug

What did you do?

We’re using traefik as a gRPC proxy and were running load tests when it crashed. We’ve recently (today) enabled access logs. We’re not able to reliably reproduce the issue but I’m guessing that the cause is pretty apparent from the crash dump below. Let me know if any more information is required.

What did you expect to see?

traefik should not crash

What did you see instead?

traefik crashed.

Output of traefik version: (What version of Traefik are you using?)

Version:      2.0.4
Codename:     montdor
Go version:   go1.13.3
Built:        2019-10-28T20:23:57Z
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, …)?

[entryPoints]
  [entryPoints.grpc]
    address = ":443"

[api]

[accessLog]
  bufferingSize = 100

[log]

[providers.file]
  filename = "/var/traefik/dynamic.toml"

dynamic.toml

[[tls.certificates]]
  certFile = "/var/pki/tls.crt"
  keyFile = "/var/pki/tls.key"

[tls.stores]
  [tls.stores.default]
    [tls.stores.default.defaultCertificate]
      certFile = "/var/pki/tls.crt"
      keyFile = "/var/pki/tls.key"

[tls.options]
  [tls.options.default]
    [tls.options.default.clientAuth]
      caFiles = ["/var/pki/ca.crt"]
      clientAuthType = "RequireAndVerifyClientCert"

[http]
  [http.routers]
    [http.routers.login]
      entryPoints = ["grpc"]
      service = "login"
      rule = "PathPrefix(`/login`)"
      [http.routers.login.tls]
 # a couple more routers like the one above


  [http.services]
    [http.services.login]
      [http.services.login.loadBalancer]
        [[http.services.login.loadBalancer.servers]]
          url = "h2c://login:7070"
    # 5 more services similar to login

Crash Dump

fatal error: concurrent map iteration and map write

goroutine 105 [running]:
runtime.throw(0x24f6109, 0x26)
	/usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc002f29b60 sp=0xc002f29b30 pc=0x42f712
runtime.mapiternext(0xc002f29c48)
	/usr/local/go/src/runtime/map.go:858 +0x579 fp=0xc002f29be8 sp=0xc002f29b60 pc=0x40f609
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*Handler).redactHeaders(0xc0005b9260, 0xc0012cd500, 0xc002f29da0, 0x24b750f, 0xb)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:319 +0x85 fp=0xc002f29cb8 sp=0xc002f29be8 pc=0x1cf0b25
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*Handler).logTheRoundTrip(0xc0005b9260, 0xc000717040, 0xc000717060, 0xc000717080)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:310 +0x8c7 fp=0xc002f29f50 sp=0xc002f29cb8 pc=0x1cf07a7
github.com/containous/traefik/v2/pkg/middlewares/accesslog.NewHandler.func1(0xc0005b9260)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:125 +0xc2 fp=0xc002f29fd8 sp=0xc002f29f50 pc=0x1cf1f02
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc002f29fe0 sp=0xc002f29fd8 pc=0x45f571
created by github.com/containous/traefik/v2/pkg/middlewares/accesslog.NewHandler
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:122 +0x265

goroutine 1 [chan receive, 27 minutes]:
github.com/containous/traefik/v2/pkg/server.(*Server).Wait(...)
	/go/src/github.com/containous/traefik/pkg/server/server.go:193
main.runCmd(0xc000490cb0, 0x0, 0x0)
	/go/src/github.com/containous/traefik/cmd/traefik/traefik.go:185 +0xe85
main.main.func1(0xc000116160, 0x2, 0x2, 0x2, 0xc000490d20)
	/go/src/github.com/containous/traefik/cmd/traefik/traefik.go:49 +0x2c
github.com/containous/traefik/v2/pkg/cli.run(0xc000490d20, 0xc000116160, 0x2, 0x2, 0xc, 0xc0005c2300)
	/go/src/github.com/containous/traefik/pkg/cli/commands.go:117 +0x171
github.com/containous/traefik/v2/pkg/cli.execute(0xc000490d20, 0xc000116150, 0x3, 0x3, 0x43d501, 0x3bc64a0, 0xc0000ce750)
	/go/src/github.com/containous/traefik/pkg/cli/commands.go:56 +0x5a2
github.com/containous/traefik/v2/pkg/cli.Execute(...)
	/go/src/github.com/containous/traefik/pkg/cli/commands.go:40
main.main()
	/go/src/github.com/containous/traefik/cmd/traefik/traefik.go:65 +0x426

goroutine 34 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0001b4820)
	/go/pkg/mod/go.opencensus.io@v0.21.0/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
	/go/pkg/mod/go.opencensus.io@v0.21.0/stats/view/worker.go:32 +0x57

goroutine 7 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x3f0dec0)
	/go/pkg/mod/k8s.io/klog@v0.3.1/klog.go:990 +0x8b
created by k8s.io/klog.init.0
	/go/pkg/mod/k8s.io/klog@v0.3.1/klog.go:404 +0x6c

goroutine 8 [syscall, 27 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 100 [select, 27 minutes]:
io.(*pipe).Read(0xc00013f720, 0xc0001ab000, 0x1000, 0x1000, 0x1f87020, 0xc00024ce01, 0xc0001ab000)
	/usr/local/go/src/io/pipe.go:50 +0xe7
io.(*PipeReader).Read(0xc0001386b8, 0xc0001ab000, 0x1000, 0x1000, 0xc00024ce48, 0x4054e5, 0x43153c)
	/usr/local/go/src/io/pipe.go:127 +0x4c
bufio.(*Scanner).Scan(0xc0002f3f38, 0xc0001b8180)
	/usr/local/go/src/bufio/scan.go:213 +0xa4
github.com/sirupsen/logrus.(*Entry).writerScanner(0xc000480230, 0xc0001386b8, 0xc0005c3030)
	/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/writer.go:53 +0xc3
created by github.com/sirupsen/logrus.(*Entry).WriterLevel
	/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/writer.go:45 +0x1a5

goroutine 99 [select, 27 minutes]:
io.(*pipe).Read(0xc00013e140, 0xc0001aa000, 0x1000, 0x1000, 0x1f87020, 0x1, 0xc0001aa000)
	/usr/local/go/src/io/pipe.go:50 +0xe7
io.(*PipeReader).Read(0xc000138100, 0xc0001aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:127 +0x4c
bufio.(*Scanner).Scan(0xc0002f2f38, 0xc00013b138)
	/usr/local/go/src/bufio/scan.go:213 +0xa4
github.com/sirupsen/logrus.(*Entry).writerScanner(0xc000490c40, 0xc000138100, 0xc0005c2390)
	/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/writer.go:53 +0xc3
created by github.com/sirupsen/logrus.(*Entry).WriterLevel
	/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/writer.go:45 +0x1a5

goroutine 101 [chan receive, 17 minutes]:
main.checkNewVersion.func1()
	/go/src/github.com/containous/traefik/cmd/traefik/traefik.go:281 +0x55
github.com/containous/traefik/v2/pkg/safe.GoWithRecover.func1(0x25ab3a0, 0xc00036a710)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:153 +0x57
created by github.com/containous/traefik/v2/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:147 +0x49

goroutine 102 [chan receive, 27 minutes]:
github.com/containous/traefik/v2/pkg/server.(*httpForwarder).Accept(0xc00044d000, 0x22c54a0, 0xc000324600, 0x20607c0, 0x3ee5520)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:46 +0x41
net/http.(*Server).Serve(0xc00048e000, 0x2909000, 0xc00044d000, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2896 +0x286
github.com/containous/traefik/v2/pkg/server.createHTTPServer.func1(0xc00048e000, 0xc00044d000, 0x2921d80, 0xc0003604b0)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:408 +0x45
created by github.com/containous/traefik/v2/pkg/server.createHTTPServer
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:407 +0x29f

goroutine 103 [chan receive, 13 minutes]:
github.com/containous/traefik/v2/pkg/server.(*httpForwarder).Accept(0xc00044d040, 0xc0002eedf8, 0x18, 0xc000561380, 0x71301e)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:46 +0x41
net/http.(*Server).Serve(0xc00048e0e0, 0x2909000, 0xc00044d040, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2896 +0x286
github.com/containous/traefik/v2/pkg/server.createHTTPServer.func1(0xc00048e0e0, 0xc00044d040, 0x2921d80, 0xc0003604b0)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:408 +0x45
created by github.com/containous/traefik/v2/pkg/server.createHTTPServer
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:407 +0x29f

goroutine 85 [chan receive, 27 minutes]:
github.com/containous/traefik/v2/cmd.ContextWithSignal.func1(0xc0001108a0, 0xc00036a8c0)
	/go/src/github.com/containous/traefik/cmd/context.go:17 +0x39
created by github.com/containous/traefik/v2/cmd.ContextWithSignal
	/go/src/github.com/containous/traefik/cmd/context.go:15 +0x104

goroutine 86 [chan receive, 27 minutes]:
github.com/containous/traefik/v2/pkg/server.(*Server).Start.func1(0xc00049c000, 0x2921cc0, 0xc000467240)
	/go/src/github.com/containous/traefik/pkg/server/server.go:171 +0x86
created by github.com/containous/traefik/v2/pkg/server.(*Server).Start
	/go/src/github.com/containous/traefik/pkg/server/server.go:169 +0x57

goroutine 147 [select, 1 minutes]:
net/http.(*http2serverConn).serve(0xc00054c900)
	/usr/local/go/src/net/http/h2_bundle.go:4423 +0x588
net/http.(*http2Server).ServeConn(0xc0005ca880, 0x2941480, 0xc00054aa80, 0xc00033bc10)
	/usr/local/go/src/net/http/h2_bundle.go:4031 +0x73a
net/http.http2ConfigureServer.func1(0xc00048e0e0, 0xc00054aa80, 0x28c87a0, 0xc00012f740)
	/usr/local/go/src/net/http/h2_bundle.go:3857 +0xef
net/http.(*conn).serve(0xc00046a0a0, 0x2921d80, 0xc00048c810)
	/usr/local/go/src/net/http/server.go:1800 +0x122c
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2927 +0x38e

goroutine 12 [IO wait, 13 minutes]:
internal/poll.runtime_pollWait(0x7fce3e8ebea8, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0005cc998, 0x72, 0x0, 0x0, 0x24af21f)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0005cc980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8
net.(*netFD).accept(0xc0005cc980, 0xb4, 0x0, 0x20)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc00044cfc0, 0x45c930, 0xc0005ebf30, 0xc0005ebf38)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).AcceptTCP(0xc00044cfc0, 0x10, 0x25ab390, 0x25ab3a0)
	/usr/local/go/src/net/tcpsock.go:248 +0x47
github.com/containous/traefik/v2/pkg/server.tcpKeepAliveListener.Accept(0xc00044cfc0, 0x25ab3a0, 0x2949b20, 0xc00019a520, 0x0)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:241 +0x2f
github.com/containous/traefik/v2/pkg/server.(*TCPEntryPoint).startTCP(0xc00013f9a0, 0x2921d80, 0xc00046cc00)
	/go/src/github.com/containous/traefik/pkg/server/server_entrypoint_tcp.go:135 +0xad
created by github.com/containous/traefik/v2/pkg/server.(*Server).startTCPServers
	/go/src/github.com/containous/traefik/pkg/server/server.go:258 +0x275

goroutine 13 [select, 27 minutes]:
github.com/containous/traefik/v2/pkg/server.(*Server).listenProviders(0xc00049c000, 0xc00026e000)
	/go/src/github.com/containous/traefik/pkg/server/server.go:264 +0xc8
github.com/containous/traefik/v2/pkg/server.(*Server).Start.func2(0xc00026e000)
	/go/src/github.com/containous/traefik/pkg/server/server.go:180 +0x34
github.com/containous/traefik/v2/pkg/safe.(*Pool).Go.func1()
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:90 +0x7b
github.com/containous/traefik/v2/pkg/safe.GoWithRecover.func1(0x25ab3a0, 0xc00046cc30)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:153 +0x57
created by github.com/containous/traefik/v2/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:147 +0x49

goroutine 14 [select, 27 minutes]:
github.com/containous/traefik/v2/pkg/server.(*Server).listenConfigurations(0xc00049c000, 0xc00026e0e0)
	/go/src/github.com/containous/traefik/pkg/server/server_configuration.go:242 +0xe1
github.com/containous/traefik/v2/pkg/server.(*Server).Start.func3(0xc00026e0e0)
	/go/src/github.com/containous/traefik/pkg/server/server.go:183 +0x34
github.com/containous/traefik/v2/pkg/safe.(*Pool).Go.func1()
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:90 +0x7b
github.com/containous/traefik/v2/pkg/safe.GoWithRecover.func1(0x25ab3a0, 0xc00046cc60)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:153 +0x57
created by github.com/containous/traefik/v2/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:147 +0x49

goroutine 91 [select, 27 minutes]:
github.com/containous/traefik/v2/pkg/server.(*Server).throttleProviderConfigReload(0xc00049c000, 0x77359400, 0xc0005b9020, 0xc0001b85a0, 0xc00023a230)
	/go/src/github.com/containous/traefik/pkg/server/server_configuration.go:277 +0x214
github.com/containous/traefik/v2/pkg/server.(*Server).preLoadConfiguration.func1(0xc00023a230)
	/go/src/github.com/containous/traefik/pkg/server/server_configuration.go:229 +0x52
github.com/containous/traefik/v2/pkg/safe.(*Pool).Go.func1()
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:90 +0x7b
github.com/containous/traefik/v2/pkg/safe.GoWithRecover.func1(0x25ab3a0, 0xc000360d20)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:153 +0x57
created by github.com/containous/traefik/v2/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:147 +0x49

goroutine 16 [select, 27 minutes]:
github.com/containous/traefik/v2/pkg/server.(*Server).listenSignals(0xc00049c000, 0xc00026e150)
	/go/src/github.com/containous/traefik/pkg/server/server_signals.go:18 +0xce
github.com/containous/traefik/v2/pkg/server.(*Server).Start.func4(0xc00026e150)
	/go/src/github.com/containous/traefik/pkg/server/server.go:187 +0x34
github.com/containous/traefik/v2/pkg/safe.(*Pool).Go.func1()
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:90 +0x7b
github.com/containous/traefik/v2/pkg/safe.GoWithRecover.func1(0x25ab3a0, 0xc00046cd50)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:153 +0x57
created by github.com/containous/traefik/v2/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:147 +0x49

goroutine 89 [syscall, 27 minutes]:
syscall.Syscall6(0xe8, 0x6, 0xc00064fbec, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
golang.org/x/sys/unix.EpollWait(0x6, 0xc00064fbec, 0x7, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/sys@v0.0.0-20190813064441-fde4db37ae7a/unix/zsyscall_linux_amd64.go:1760 +0x72
gopkg.in/fsnotify%2ev1.(*fdPoller).wait(0xc0004d4280, 0x0, 0x0, 0x0)
	/go/pkg/mod/gopkg.in/fsnotify.v1@v1.4.7/inotify_poller.go:86 +0x91
gopkg.in/fsnotify%2ev1.(*Watcher).readEvents(0xc00046e870)
	/go/pkg/mod/gopkg.in/fsnotify.v1@v1.4.7/inotify.go:192 +0x1f8
created by gopkg.in/fsnotify%2ev1.NewWatcher
	/go/pkg/mod/gopkg.in/fsnotify.v1@v1.4.7/inotify.go:59 +0x1a5

goroutine 90 [select, 27 minutes]:
github.com/containous/traefik/v2/pkg/provider/file.(*Provider).addWatcher.func1(0xc00023a1c0)
	/go/src/github.com/containous/traefik/pkg/provider/file/file.go:109 +0x183
github.com/containous/traefik/v2/pkg/safe.(*Pool).Go.func1()
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:90 +0x7b
github.com/containous/traefik/v2/pkg/safe.GoWithRecover.func1(0x25ab3a0, 0xc000360c90)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:153 +0x57
created by github.com/containous/traefik/v2/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:147 +0x49

goroutine 92 [select, 27 minutes]:
github.com/eapache/channels.(*RingChannel).ringBuffer(0xc000360d80)
	/go/pkg/mod/github.com/eapache/channels@v1.1.0/ring_channel.go:87 +0x1ff
created by github.com/eapache/channels.NewRingChannel
	/go/pkg/mod/github.com/eapache/channels@v1.1.0/ring_channel.go:32 +0x1b3

goroutine 93 [select, 27 minutes]:
github.com/containous/traefik/v2/pkg/server.(*Server).throttleProviderConfigReload.func1(0xc00023a2a0)
	/go/src/github.com/containous/traefik/pkg/server/server_configuration.go:264 +0xd9
github.com/containous/traefik/v2/pkg/safe.(*Pool).Go.func1()
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:90 +0x7b
github.com/containous/traefik/v2/pkg/safe.GoWithRecover.func1(0x25ab3a0, 0xc000360db0)
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:153 +0x57
created by github.com/containous/traefik/v2/pkg/safe.GoWithRecover
	/go/src/github.com/containous/traefik/pkg/safe/routine.go:147 +0x49

goroutine 145 [select, 7 minutes]:
github.com/patrickmn/go-cache.(*janitor).Run(0xc0005c2b00, 0xc00013d640)
	/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079 +0xd5
created by github.com/patrickmn/go-cache.runJanitor
	/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0xa9

goroutine 114 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fce3e8ebdd8, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0005cc218, 0x72, 0x800, 0x817, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0005cc200, 0xc000154900, 0x817, 0x817, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0005cc200, 0xc000154900, 0x817, 0x817, 0x414557, 0x294a5294a5, 0x37)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00019a098, 0xc000154900, 0x817, 0x817, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x68
github.com/containous/traefik/v2/pkg/tcp.(*Conn).Read(0xc0004be090, 0xc000154900, 0x817, 0x817, 0xc000109a40, 0x4461ce, 0x10)
	/go/src/github.com/containous/traefik/pkg/tcp/router.go:173 +0x134
crypto/tls.(*atLeastReader).Read(0xc0007824c0, 0xc000154900, 0x817, 0x817, 0xc0001240f0, 0xc000109b18, 0xc000109a78)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc00054acd8, 0x28c1420, 0xc0007824c0, 0x40a2e5, 0x20d88c0, 0x22c01a0)
	/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc00054aa80, 0x7fce3e6f8950, 0xc0004be090, 0x5, 0xc0004be090, 0xc000109b38)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc00054aa80, 0x0, 0x0, 0x6d442b)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc00054aa80, 0xc00048e3b8, 0x9, 0x9, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
io.ReadAtLeast(0x7fce3e6f8a48, 0xc00054aa80, 0xc00048e3b8, 0x9, 0x9, 0x9, 0x170f04d099701, 0x0, 0x1)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
net/http.http2readFrameHeader(0xc00048e3b8, 0x9, 0x9, 0x7fce3e6f8a48, 0xc00054aa80, 0x0, 0xc000000000, 0xc000109f28, 0xc0001b8180)
	/usr/local/go/src/net/http/h2_bundle.go:1477 +0x87
net/http.(*http2Framer).ReadFrame(0xc00048e380, 0xc000109ee0, 0x2, 0x0, 0x1)
	/usr/local/go/src/net/http/h2_bundle.go:1735 +0xa1
net/http.(*http2serverConn).readFrames(0xc00054c900)
	/usr/local/go/src/net/http/h2_bundle.go:4310 +0xa4
created by net/http.(*http2serverConn).serve
	/usr/local/go/src/net/http/h2_bundle.go:4415 +0x347

goroutine 280 [IO wait]:
internal/poll.runtime_pollWait(0x7fce3e8eba98, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00019ea18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00019ea00, 0xc00075e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc00019ea00, 0xc00075e000, 0x1000, 0x1000, 0xba9452, 0xc000aa2601, 0xc0005e6c60)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0001384c0, 0xc00075e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc0005b98c0, 0xc0006bc8f8, 0x9, 0x9, 0xc0005e6d10, 0xba57ed, 0xc0008ea650)
	/usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x28c1220, 0xc0005b98c0, 0xc0006bc8f8, 0x9, 0x9, 0x9, 0xc000aa2640, 0xc0005e6d38, 0xba56f8)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0006bc8f8, 0x9, 0x9, 0x28c1220, 0xc0005b98c0, 0x0, 0x0, 0xc001272090, 0x10)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0006bc8c0, 0xc0012720c0, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0005e6fb8, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1710 +0x8c
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0008ea600)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1638 +0xa3
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:673 +0x62f

goroutine 125 [IO wait]:
internal/poll.runtime_pollWait(0x7fce3e8ebc38, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00056a918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00056a900, 0xc0004ee000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc00056a900, 0xc0004ee000, 0x1000, 0x1000, 0xba9452, 0xc000618d01, 0xc00292ec60)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000e088, 0xc0004ee000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc00013aa20, 0xc0001ee1f8, 0x9, 0x9, 0xc00292ed10, 0xba57ed, 0xc0005a70d0)
	/usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x28c1220, 0xc00013aa20, 0xc0001ee1f8, 0x9, 0x9, 0x9, 0xc000618dc0, 0xc00292ed38, 0xba56f8)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0001ee1f8, 0x9, 0x9, 0x28c1220, 0xc00013aa20, 0x0, 0x0, 0xc000eef500, 0x10)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001ee1c0, 0xc000eef530, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00292efb8, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1710 +0x8c
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0005a7080)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1638 +0xa3
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:673 +0x62f

goroutine 176 [IO wait, 24 minutes]:
internal/poll.runtime_pollWait(0x7fce3e8ebb68, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00019e618, 0x72, 0x800, 0x817, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00019e600, 0xc0006fa000, 0x817, 0x817, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc00019e600, 0xc0006fa000, 0x817, 0x817, 0x0, 0x80a, 0xc00044d7c0)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0005bc010, 0xc0006fa000, 0x817, 0x817, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x68
github.com/containous/traefik/v2/pkg/tcp.(*Conn).Read(0xc0003681e0, 0xc0006fa000, 0x817, 0x817, 0xc00052fa40, 0x4461ce, 0x10)
	/go/src/github.com/containous/traefik/pkg/tcp/router.go:173 +0x134
crypto/tls.(*atLeastReader).Read(0xc0005c7420, 0xc0006fa000, 0x817, 0x817, 0xc00047ec10, 0xc00052fb18, 0xc00052fa78)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc00055e958, 0x28c1420, 0xc0005c7420, 0x40a2e5, 0x20d88c0, 0x22c01a0)
	/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc00055e700, 0x7fce3e6f8950, 0xc0003681e0, 0x5, 0xc0003681e0, 0xc00052fb38)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc00055e700, 0x0, 0x0, 0x6d442b)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc00055e700, 0xc0001ee498, 0x9, 0x9, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
io.ReadAtLeast(0x7fce3e6f8a48, 0xc00055e700, 0xc0001ee498, 0x9, 0x9, 0x9, 0x170f04d099701, 0x0, 0x1)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
net/http.http2readFrameHeader(0xc0001ee498, 0x9, 0x9, 0x7fce3e6f8a48, 0xc00055e700, 0x0, 0xc000000000, 0xc00052ff28, 0xc0004dc360)
	/usr/local/go/src/net/http/h2_bundle.go:1477 +0x87
net/http.(*http2Framer).ReadFrame(0xc0001ee460, 0xc00052fee0, 0x2, 0x0, 0x1)
	/usr/local/go/src/net/http/h2_bundle.go:1735 +0xa1
net/http.(*http2serverConn).readFrames(0xc0005a7200)
	/usr/local/go/src/net/http/h2_bundle.go:4310 +0xa4
created by net/http.(*http2serverConn).serve
	/usr/local/go/src/net/http/h2_bundle.go:4415 +0x347

goroutine 177 [select, 24 minutes]:
net/http.(*http2serverConn).serve(0xc000580c00)
	/usr/local/go/src/net/http/h2_bundle.go:4423 +0x588
net/http.(*http2Server).ServeConn(0xc0005ca880, 0x2941480, 0xc000592a80, 0xc0006bbc10)
	/usr/local/go/src/net/http/h2_bundle.go:4031 +0x73a
net/http.http2ConfigureServer.func1(0xc00048e0e0, 0xc000592a80, 0x28c87a0, 0xc0005c7100)
	/usr/local/go/src/net/http/h2_bundle.go:3857 +0xef
net/http.(*conn).serve(0xc0005d20a0, 0x2921d80, 0xc00048c810)
	/usr/local/go/src/net/http/server.go:1800 +0x122c
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2927 +0x38e

goroutine 43 [select, 24 minutes]:
net/http.(*http2serverConn).serve(0xc0005a7200)
	/usr/local/go/src/net/http/h2_bundle.go:4423 +0x588
net/http.(*http2Server).ServeConn(0xc0005ca880, 0x2941480, 0xc00055e700, 0xc0006b7c10)
	/usr/local/go/src/net/http/h2_bundle.go:4031 +0x73a
net/http.http2ConfigureServer.func1(0xc00048e0e0, 0xc00055e700, 0x28c87a0, 0xc00044d800)
	/usr/local/go/src/net/http/h2_bundle.go:3857 +0xef
net/http.(*conn).serve(0xc0004921e0, 0x2921d80, 0xc00048c810)
	/usr/local/go/src/net/http/server.go:1800 +0x122c
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2927 +0x38e

goroutine 44 [IO wait, 24 minutes]:
internal/poll.runtime_pollWait(0x7fce3e8ebd08, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00019e418, 0x72, 0x800, 0x817, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00019e400, 0xc0006ae000, 0x817, 0x817, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc00019e400, 0xc0006ae000, 0x817, 0x817, 0x0, 0x80a, 0xc0005c70c0)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0005bc008, 0xc0006ae000, 0x817, 0x817, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x68
github.com/containous/traefik/v2/pkg/tcp.(*Conn).Read(0xc000324d80, 0xc0006ae000, 0x817, 0x817, 0xc0005e5a40, 0x4461ce, 0x10)
	/go/src/github.com/containous/traefik/pkg/tcp/router.go:173 +0x134
crypto/tls.(*atLeastReader).Read(0xc0001a0740, 0xc0006ae000, 0x817, 0x817, 0xc00019dfb0, 0xc0005e5b18, 0xc0005e5a78)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc000592cd8, 0x28c1420, 0xc0001a0740, 0x40a2e5, 0x20d88c0, 0x22c01a0)
	/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc000592a80, 0x7fce3e6f8950, 0xc000324d80, 0x5, 0xc000324d80, 0xc0005e5b38)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc000592a80, 0x0, 0x0, 0x6d442b)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc000592a80, 0xc0006bc1f8, 0x9, 0x9, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
io.ReadAtLeast(0x7fce3e6f8a48, 0xc000592a80, 0xc0006bc1f8, 0x9, 0x9, 0x9, 0x170f04d099701, 0x0, 0x1)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
net/http.http2readFrameHeader(0xc0006bc1f8, 0x9, 0x9, 0x7fce3e6f8a48, 0xc000592a80, 0x0, 0xc000000000, 0xc0005e5f28, 0xc0004dc300)
	/usr/local/go/src/net/http/h2_bundle.go:1477 +0x87
net/http.(*http2Framer).ReadFrame(0xc0006bc1c0, 0xc0005e5ee0, 0x2, 0x0, 0x1)
	/usr/local/go/src/net/http/h2_bundle.go:1735 +0xa1
net/http.(*http2serverConn).readFrames(0xc000580c00)
	/usr/local/go/src/net/http/h2_bundle.go:4310 +0xa4
created by net/http.(*http2serverConn).serve
	/usr/local/go/src/net/http/h2_bundle.go:4415 +0x347

goroutine 107 [IO wait]:
internal/poll.runtime_pollWait(0x7fce3e8eb8f8, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0005cc198, 0x72, 0x5b00, 0x5b14, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0005cc180, 0xc001b6a000, 0x5b14, 0x5b14, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0005cc180, 0xc001b6a000, 0x5b14, 0x5b14, 0x1, 0xc000c63288, 0xb1572ae440cb7001)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00019a008, 0xc001b6a000, 0x5b14, 0x5b14, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x68
github.com/containous/traefik/v2/pkg/tcp.(*Conn).Read(0xc00048c3c0, 0xc001b6a000, 0x5b14, 0x5b14, 0xdf98a167b388a327, 0xc00052ea68, 0x10)
	/go/src/github.com/containous/traefik/pkg/tcp/router.go:173 +0x134
crypto/tls.(*atLeastReader).Read(0xc003f292a0, 0xc001b6a000, 0x5b14, 0x5b14, 0xc000848475, 0x3, 0xc00052ea78)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc000593ad8, 0x28c1420, 0xc003f292a0, 0x40a2e5, 0x20d88c0, 0x22c01a0)
	/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc000593880, 0x7fce3e6f8950, 0xc00048c3c0, 0x5, 0xc00048c3c0, 0xc00052eb38)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc000593880, 0x0, 0x0, 0x6d442b)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc000593880, 0xc0006bc658, 0x9, 0x9, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
io.ReadAtLeast(0x7fce3e6f8a48, 0xc000593880, 0xc0006bc658, 0x9, 0x9, 0x9, 0x170f04d099701, 0x0, 0x1)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
net/http.http2readFrameHeader(0xc0006bc658, 0x9, 0x9, 0x7fce3e6f8a48, 0xc000593880, 0x0, 0xc000000000, 0xc00052ef28, 0xc0004de000)
	/usr/local/go/src/net/http/h2_bundle.go:1477 +0x87
net/http.(*http2Framer).ReadFrame(0xc0006bc620, 0xc00052eee0, 0x2, 0x0, 0x1)
	/usr/local/go/src/net/http/h2_bundle.go:1735 +0xa1
net/http.(*http2serverConn).readFrames(0xc000595980)
	/usr/local/go/src/net/http/h2_bundle.go:4310 +0xa4
created by net/http.(*http2serverConn).serve
	/usr/local/go/src/net/http/h2_bundle.go:4415 +0x347

goroutine 239 [select]:
net/http.(*http2serverConn).serve(0xc000595980)
	/usr/local/go/src/net/http/h2_bundle.go:4423 +0x588
net/http.(*http2Server).ServeConn(0xc0005ca880, 0x2941480, 0xc000593880, 0xc000509c10)
	/usr/local/go/src/net/http/h2_bundle.go:4031 +0x73a
net/http.http2ConfigureServer.func1(0xc00048e0e0, 0xc000593880, 0x28c87a0, 0xc00000c680)
	/usr/local/go/src/net/http/h2_bundle.go:3857 +0xef
net/http.(*conn).serve(0xc000492000, 0x2921d80, 0xc00048c810)
	/usr/local/go/src/net/http/server.go:1800 +0x122c
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2927 +0x38e

goroutine 315 [IO wait]:
internal/poll.runtime_pollWait(0x7fce3e8eb828, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0005cd018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0005cd000, 0xc000823000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0005cd000, 0xc000823000, 0x1000, 0x1000, 0xba9452, 0xc000df9101, 0xc000530c60)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00019a1e0, 0xc000823000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc00077a600, 0xc00048e498, 0x9, 0x9, 0xc000530d10, 0xba57ed, 0xc0007e41d0)
	/usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x28c1220, 0xc00077a600, 0xc00048e498, 0x9, 0x9, 0x9, 0xc000df9180, 0xc000530d38, 0xba56f8)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc00048e498, 0x9, 0x9, 0x28c1220, 0xc00077a600, 0x0, 0x0, 0xc000a5c0f0, 0x10)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00048e460, 0xc000a5c120, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000530fb8, 0x0, 0xc00054b518)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1710 +0x8c
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0007e4180)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1638 +0xa3
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:673 +0x62f

goroutine 533 [IO wait]:
internal/poll.runtime_pollWait(0x7fce3e8eb758, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0009d4998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0009d4980, 0xc000a54000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0009d4980, 0xc000a54000, 0x1000, 0x1000, 0x65676173, 0xc001146bc0, 0xc0010f11e0)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0009e2028, 0xc000a54000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x68
bufio.(*Reader).Read(0xc0009a2240, 0xc0001ee2d8, 0x9, 0x9, 0xc0002f1d10, 0xba57ed, 0xc000a52050)
	/usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x28c1220, 0xc0009a2240, 0xc0001ee2d8, 0x9, 0x9, 0x9, 0xc000b4b7c0, 0xc0002f1d38, 0xba56f8)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0001ee2d8, 0x9, 0x9, 0x28c1220, 0xc0009a2240, 0x0, 0x0, 0xc001274780, 0x10)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001ee2a0, 0xc0012747b0, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0002f1fb8, 0x5be1de, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1710 +0x8c
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000a52000)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1638 +0xa3
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:673 +0x62f

goroutine 894 [select]:
net/http.(*http2serverConn).serve(0xc000a52d80)
	/usr/local/go/src/net/http/h2_bundle.go:4423 +0x588
net/http.(*http2Server).ServeConn(0xc0005ca880, 0x2941480, 0xc00054bc00, 0xc000b27c10)
	/usr/local/go/src/net/http/h2_bundle.go:4031 +0x73a
net/http.http2ConfigureServer.func1(0xc00048e0e0, 0xc00054bc00, 0x28c87a0, 0xc0003642a0)
	/usr/local/go/src/net/http/h2_bundle.go:3857 +0xef
net/http.(*conn).serve(0xc0005b61e0, 0x2921d80, 0xc00048c810)
	/usr/local/go/src/net/http/server.go:1800 +0x122c
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2927 +0x38e

goroutine 913 [IO wait]:
internal/poll.runtime_pollWait(0x7fce3e8eb9c8, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00056bd98, 0x72, 0x9200, 0x92e1, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00056bd80, 0xc00136c000, 0x92e1, 0x92e1, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc00056bd80, 0xc00136c000, 0x92e1, 0x92e1, 0x2, 0xc000df1a88, 0xfc1608817c0ae01)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00019a2f0, 0xc00136c000, 0x92e1, 0x92e1, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x68
github.com/containous/traefik/v2/pkg/tcp.(*Conn).Read(0xc000b1cc90, 0xc00136c000, 0x92e1, 0x92e1, 0x8628833a5f13ce29, 0xc000b10a68, 0x10)
	/go/src/github.com/containous/traefik/pkg/tcp/router.go:173 +0x134
crypto/tls.(*atLeastReader).Read(0xc003f296e0, 0xc00136c000, 0x92e1, 0x92e1, 0xc0007190fd, 0x3, 0xc000b10a78)
	/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc00054be58, 0x28c1420, 0xc003f296e0, 0x40a2e5, 0x20d88c0, 0x22c01a0)
	/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc00054bc00, 0x7fce3e6f8950, 0xc000b1cc90, 0x5, 0xc000b1cc90, 0xc000b10b38)
	/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc00054bc00, 0x0, 0x0, 0x6d442b)
	/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc00054bc00, 0xc00048e818, 0x9, 0x9, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
io.ReadAtLeast(0x7fce3e6f8a48, 0xc00054bc00, 0xc00048e818, 0x9, 0x9, 0x9, 0x170f04d099701, 0x0, 0x1)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
net/http.http2readFrameHeader(0xc00048e818, 0x9, 0x9, 0x7fce3e6f8a48, 0xc00054bc00, 0x0, 0xc000000000, 0xc000b10f28, 0xc0008660c0)
	/usr/local/go/src/net/http/h2_bundle.go:1477 +0x87
net/http.(*http2Framer).ReadFrame(0xc00048e7e0, 0xc000b10ee0, 0x2, 0x0, 0x1)
	/usr/local/go/src/net/http/h2_bundle.go:1735 +0xa1
net/http.(*http2serverConn).readFrames(0xc000a52d80)
	/usr/local/go/src/net/http/h2_bundle.go:4310 +0xa4
created by net/http.(*http2serverConn).serve
	/usr/local/go/src/net/http/h2_bundle.go:4415 +0x347

goroutine 4843 [select]:
net/http.(*http2serverConn).readFrames(0xc000f05b00)
	/usr/local/go/src/net/http/h2_bundle.go:4311 +0x181
created by net/http.(*http2serverConn).serve
	/usr/local/go/src/net/http/h2_bundle.go:4415 +0x347

goroutine 658876 [select]:
golang.org/x/net/http2.(*ClientConn).roundTrip(0xc0007e4180, 0xc000d6c700, 0x0, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1084 +0x9a0
golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0005b9140, 0xc000d6c700, 0xc00035af00, 0x3f2b9e8, 0xc00035afc8, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:447 +0x172
golang.org/x/net/http2.(*Transport).RoundTrip(...)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:408
github.com/containous/traefik/v2/pkg/server.(*h2cTransportWrapper).RoundTrip(0xc000138ef0, 0xc000d6c700, 0xc00011a4e0, 0x3, 0xc00035b048)
	/go/src/github.com/containous/traefik/pkg/server/roundtripper.go:23 +0x60
net/http.(*Transport).roundTrip(0xc0001d3a40, 0xc000d6c700, 0x1f9fea0, 0x1, 0xc001046840)
	/usr/local/go/src/net/http/transport.go:485 +0xd87
net/http.(*Transport).RoundTrip(0xc0001d3a40, 0xc000d6c700, 0x24bf627, 0xf, 0xc001046928)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x35
net/http/httputil.(*ReverseProxy).ServeHTTP(0xc0005d0410, 0x2908e80, 0xc0006f8240, 0xc000d6c600)
	/usr/local/go/src/net/http/httputil/reverseproxy.go:249 +0x2a4
github.com/containous/traefik/v2/pkg/middlewares/pipelining.(*pipelining).ServeHTTP(0xc0001983b0, 0x2908e80, 0xc0006f8240, 0xc000d6c600)
	/go/src/github.com/containous/traefik/pkg/middlewares/pipelining/pipelining.go:34 +0x6f
github.com/containous/traefik/v2/pkg/middlewares/accesslog.AddServiceFields(0x2908e80, 0xc0006f8240, 0xc000d6c600, 0x28c21a0, 0xc0001983b0, 0xc0006f81a0)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:47 +0x134
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc00013db00, 0x2908e80, 0xc0006f8240, 0xc000d6c600)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0x10b
github.com/vulcand/oxy/roundrobin.(*RoundRobin).ServeHTTP(0xc000488af0, 0x2908e80, 0xc0006f8240, 0xc000d6c500)
	/go/pkg/mod/github.com/vulcand/oxy@v1.0.0/roundrobin/rr.go:147 +0xfe
github.com/containous/traefik/v2/pkg/middlewares/emptybackendhandler.(*emptyBackend).ServeHTTP(0xc0001983f0, 0x2908e80, 0xc0006f8240, 0xc000d6c500)
	/go/src/github.com/containous/traefik/pkg/middlewares/emptybackendhandler/empty_backend_handler.go:31 +0x168
github.com/containous/traefik/v2/pkg/middlewares/tracing.(*forwarderMiddleware).ServeHTTP(0xc0004bf680, 0x2908e80, 0xc0006f8240, 0xc000d6c500)
	/go/src/github.com/containous/traefik/pkg/middlewares/tracing/forwarder.go:38 +0x610
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc00013dc80, 0x2908e80, 0xc0006f8240, 0xc000d6c500)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:38 +0x144
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000194a80, 0x2908e80, 0xc0006f8240, 0xc000d6c500)
	/go/pkg/mod/github.com/containous/mux@v0.0.0-20181024131434-c33f32e26898/mux.go:133 +0x104
github.com/containous/traefik/v2/pkg/middlewares/recovery.(*recovery).ServeHTTP(0xc00063c400, 0x2908e80, 0xc0006f8240, 0xc000d6c300)
	/go/src/github.com/containous/traefik/pkg/middlewares/recovery/recovery.go:33 +0x131
github.com/containous/traefik/v2/pkg/middlewares/accesslog.AddOriginFields(0x2908e80, 0xc0006f8220, 0xc000d6c300, 0x28c21e0, 0xc00063c400, 0xc0006f81a0)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:55 +0x119
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc0001a2dc0, 0x2908e80, 0xc0006f8220, 0xc000d6c300)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0x10b
github.com/containous/traefik/v2/pkg/middlewares/requestdecorator.(*RequestDecorator).ServeHTTP(0xc000138ef8, 0x2908e80, 0xc0006f8220, 0xc000d6c200, 0xc00292a9e8)
	/go/src/github.com/containous/traefik/pkg/middlewares/requestdecorator/request_decorator.go:47 +0x37d
github.com/containous/traefik/v2/pkg/middlewares/requestdecorator.WrapHandler.func1.1(0x2908e80, 0xc0006f8220, 0xc000d6c200)
	/go/src/github.com/containous/traefik/pkg/middlewares/requestdecorator/request_decorator.go:84 +0x83
net/http.HandlerFunc.ServeHTTP(0xc00063c4a0, 0x2908e80, 0xc0006f8220, 0xc000d6c200)
	/usr/local/go/src/net/http/server.go:2007 +0x44
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*Handler).ServeHTTP(0xc0005b9260, 0x2918540, 0xc001b7e008, 0xc000d6c100, 0x28c84c0, 0xc00063c4a0)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:202 +0x9e0
github.com/containous/traefik/v2/pkg/middlewares/accesslog.WrapHandler.func1.1(0x2918540, 0xc001b7e008, 0xc000d6c100)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:69 +0x5a
net/http.HandlerFunc.ServeHTTP(0xc00063c4c0, 0x2918540, 0xc001b7e008, 0xc000d6c100)
	/usr/local/go/src/net/http/server.go:2007 +0x44
github.com/containous/traefik/v2/pkg/middlewares.(*HTTPHandlerSwitcher).ServeHTTP(0xc000138ee8, 0x2918540, 0xc001b7e008, 0xc000d6c100)
	/go/src/github.com/containous/traefik/pkg/middlewares/handler_switcher.go:23 +0x70
github.com/containous/traefik/v2/pkg/middlewares/forwardedheaders.(*XForwarded).ServeHTTP(0xc00013f950, 0x2918540, 0xc001b7e008, 0xc000d6c100)
	/go/src/github.com/containous/traefik/pkg/middlewares/forwardedheaders/forwarded_header.go:174 +0x10b
net/http.serverHandler.ServeHTTP(0xc00048e0e0, 0x2918540, 0xc001b7e008, 0xc000d6c100)
	/usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.initNPNRequest.ServeHTTP(0x2921d80, 0xc000c7e840, 0xc00057f500, 0xc00048e0e0, 0x2918540, 0xc001b7e008, 0xc000d6c100)
	/usr/local/go/src/net/http/server.go:3365 +0x8d
net/http.(*http2serverConn).runHandler(0xc000f05b00, 0xc001b7e008, 0xc000d6c100, 0xc0006f8100)
	/usr/local/go/src/net/http/h2_bundle.go:5713 +0x9f
created by net/http.(*http2serverConn).processHeaders
	/usr/local/go/src/net/http/h2_bundle.go:5447 +0x4eb

goroutine 4879 [runnable]:
net/http.http2gate.Done(...)
	/usr/local/go/src/net/http/h2_bundle.go:3292
net/http.(*http2serverConn).serve(0xc000f05b00)
	/usr/local/go/src/net/http/h2_bundle.go:4436 +0xb9c
net/http.(*http2Server).ServeConn(0xc0005ca880, 0x2941480, 0xc00057f500, 0xc000923c10)
	/usr/local/go/src/net/http/h2_bundle.go:4031 +0x73a
net/http.http2ConfigureServer.func1(0xc00048e0e0, 0xc00057f500, 0x28c87a0, 0xc000717ea0)
	/usr/local/go/src/net/http/h2_bundle.go:3857 +0xef
net/http.(*conn).serve(0xc000a5a500, 0x2921d80, 0xc00048c810)
	/usr/local/go/src/net/http/server.go:1800 +0x122c
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2927 +0x38e

goroutine 659810 [runnable]:
golang.org/x/net/http2.(*ClientConn).roundTrip(0xc0007e4180, 0xc000cf0d00, 0x0, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1084 +0x9a0
golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0005b9140, 0xc000cf0d00, 0xc00035af00, 0x3f2b9e8, 0xc00035afc8, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:447 +0x172
golang.org/x/net/http2.(*Transport).RoundTrip(...)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:408
github.com/containous/traefik/v2/pkg/server.(*h2cTransportWrapper).RoundTrip(0xc000138ef0, 0xc000cf0d00, 0xc00011a4e0, 0x3, 0xc00035b048)
	/go/src/github.com/containous/traefik/pkg/server/roundtripper.go:23 +0x60
net/http.(*Transport).roundTrip(0xc0001d3a40, 0xc000cf0d00, 0x1f9fea0, 0x1, 0xc0010b62c0)
	/usr/local/go/src/net/http/transport.go:485 +0xd87
net/http.(*Transport).RoundTrip(0xc0001d3a40, 0xc000cf0d00, 0x24bf627, 0xf, 0xc0010b63d8)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x35
net/http/httputil.(*ReverseProxy).ServeHTTP(0xc0005d0410, 0x2908e80, 0xc00044cb80, 0xc000cf0c00)
	/usr/local/go/src/net/http/httputil/reverseproxy.go:249 +0x2a4
github.com/containous/traefik/v2/pkg/middlewares/pipelining.(*pipelining).ServeHTTP(0xc0001983b0, 0x2908e80, 0xc00044cb80, 0xc000cf0c00)
	/go/src/github.com/containous/traefik/pkg/middlewares/pipelining/pipelining.go:34 +0x6f
github.com/containous/traefik/v2/pkg/middlewares/accesslog.AddServiceFields(0x2908e80, 0xc00044cb80, 0xc000cf0c00, 0x28c21a0, 0xc0001983b0, 0xc00044cae0)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:47 +0x134
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc00013db00, 0x2908e80, 0xc00044cb80, 0xc000cf0c00)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0x10b
github.com/vulcand/oxy/roundrobin.(*RoundRobin).ServeHTTP(0xc000488af0, 0x2908e80, 0xc00044cb80, 0xc000cf0b00)
	/go/pkg/mod/github.com/vulcand/oxy@v1.0.0/roundrobin/rr.go:147 +0xfe
github.com/containous/traefik/v2/pkg/middlewares/emptybackendhandler.(*emptyBackend).ServeHTTP(0xc0001983f0, 0x2908e80, 0xc00044cb80, 0xc000cf0b00)
	/go/src/github.com/containous/traefik/pkg/middlewares/emptybackendhandler/empty_backend_handler.go:31 +0x168
github.com/containous/traefik/v2/pkg/middlewares/tracing.(*forwarderMiddleware).ServeHTTP(0xc0004bf680, 0x2908e80, 0xc00044cb80, 0xc000cf0b00)
	/go/src/github.com/containous/traefik/pkg/middlewares/tracing/forwarder.go:38 +0x610
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc00013dc80, 0x2908e80, 0xc00044cb80, 0xc000cf0b00)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:38 +0x144
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000194a80, 0x2908e80, 0xc00044cb80, 0xc000cf0b00)
	/go/pkg/mod/github.com/containous/mux@v0.0.0-20181024131434-c33f32e26898/mux.go:133 +0x104
github.com/containous/traefik/v2/pkg/middlewares/recovery.(*recovery).ServeHTTP(0xc00063c400, 0x2908e80, 0xc00044cb80, 0xc000cf0900)
	/go/src/github.com/containous/traefik/pkg/middlewares/recovery/recovery.go:33 +0x131
github.com/containous/traefik/v2/pkg/middlewares/accesslog.AddOriginFields(0x2908e80, 0xc00044cb60, 0xc000cf0900, 0x28c21e0, 0xc00063c400, 0xc00044cae0)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:55 +0x119
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc0001a2dc0, 0x2908e80, 0xc00044cb60, 0xc000cf0900)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0x10b
github.com/containous/traefik/v2/pkg/middlewares/requestdecorator.(*RequestDecorator).ServeHTTP(0xc000138ef8, 0x2908e80, 0xc00044cb60, 0xc000cf0800, 0xc000b119e8)
	/go/src/github.com/containous/traefik/pkg/middlewares/requestdecorator/request_decorator.go:47 +0x37d
github.com/containous/traefik/v2/pkg/middlewares/requestdecorator.WrapHandler.func1.1(0x2908e80, 0xc00044cb60, 0xc000cf0800)
	/go/src/github.com/containous/traefik/pkg/middlewares/requestdecorator/request_decorator.go:84 +0x83
net/http.HandlerFunc.ServeHTTP(0xc00063c4a0, 0x2908e80, 0xc00044cb60, 0xc000cf0800)
	/usr/local/go/src/net/http/server.go:2007 +0x44
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*Handler).ServeHTTP(0xc0005b9260, 0x2918540, 0xc00019a550, 0xc000cf0700, 0x28c84c0, 0xc00063c4a0)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:202 +0x9e0
github.com/containous/traefik/v2/pkg/middlewares/accesslog.WrapHandler.func1.1(0x2918540, 0xc00019a550, 0xc000cf0700)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:69 +0x5a
net/http.HandlerFunc.ServeHTTP(0xc00063c4c0, 0x2918540, 0xc00019a550, 0xc000cf0700)
	/usr/local/go/src/net/http/server.go:2007 +0x44
github.com/containous/traefik/v2/pkg/middlewares.(*HTTPHandlerSwitcher).ServeHTTP(0xc000138ee8, 0x2918540, 0xc00019a550, 0xc000cf0700)
	/go/src/github.com/containous/traefik/pkg/middlewares/handler_switcher.go:23 +0x70
github.com/containous/traefik/v2/pkg/middlewares/forwardedheaders.(*XForwarded).ServeHTTP(0xc00013f950, 0x2918540, 0xc00019a550, 0xc000cf0700)
	/go/src/github.com/containous/traefik/pkg/middlewares/forwardedheaders/forwarded_header.go:174 +0x10b
net/http.serverHandler.ServeHTTP(0xc00048e0e0, 0x2918540, 0xc00019a550, 0xc000cf0700)
	/usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.initNPNRequest.ServeHTTP(0x2921d80, 0xc000c7e840, 0xc00057f500, 0xc00048e0e0, 0x2918540, 0xc00019a550, 0xc000cf0700)
	/usr/local/go/src/net/http/server.go:3365 +0x8d
net/http.(*http2serverConn).runHandler(0xc000f05b00, 0xc00019a550, 0xc000cf0700, 0xc00044ca80)
	/usr/local/go/src/net/http/h2_bundle.go:5713 +0x9f
created by net/http.(*http2serverConn).processHeaders
	/usr/local/go/src/net/http/h2_bundle.go:5447 +0x4eb

goroutine 658877 [chan receive]:
net/http.http2closeWaiter.Wait(...)
	/usr/local/go/src/net/http/h2_bundle.go:3314
net/http.(*http2responseWriter).CloseNotify.func1(0xc000dc8060, 0xc00026e1c0)
	/usr/local/go/src/net/http/h2_bundle.go:6147 +0x34
created by net/http.(*http2responseWriter).CloseNotify
	/usr/local/go/src/net/http/h2_bundle.go:6146 +0xf8

goroutine 659726 [select]:
net/http.(*http2serverConn).writeHeaders(0xc000f05b00, 0xc000fb5810, 0xc0001ed500, 0xc0008adec0, 0x71546d)
	/usr/local/go/src/net/http/h2_bundle.go:5747 +0x16d
net/http.(*http2responseWriterState).writeChunk(0xc0009d4d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:6050 +0x543
net/http.(*http2responseWriter).Flush(0xc00019a548)
	/usr/local/go/src/net/http/h2_bundle.go:6131 +0x78
net/http.(*http2responseWriter).handlerDone(0xc00019a548)
	/usr/local/go/src/net/http/h2_bundle.go:6258 +0x46
net/http.(*http2serverConn).runHandler.func1(0xc00019a548, 0xc0008adf67, 0xc000f05b00)
	/usr/local/go/src/net/http/h2_bundle.go:5711 +0x2bf
net/http.(*http2serverConn).runHandler(0xc000f05b00, 0xc00019a548, 0xc000cf0500, 0xc00044c980)
	/usr/local/go/src/net/http/h2_bundle.go:5715 +0xa5
created by net/http.(*http2serverConn).processHeaders
	/usr/local/go/src/net/http/h2_bundle.go:5447 +0x4eb

goroutine 659795 [chan receive]:
net/http.http2closeWaiter.Wait(...)
	/usr/local/go/src/net/http/h2_bundle.go:3314
net/http.(*http2responseWriter).CloseNotify.func1(0xc001123260, 0xc00096eaf0)
	/usr/local/go/src/net/http/h2_bundle.go:6147 +0x34
created by net/http.(*http2responseWriter).CloseNotify
	/usr/local/go/src/net/http/h2_bundle.go:6146 +0xf8

goroutine 659820 [runnable]:
golang.org/x/net/http2.(*Transport).getBodyWriterState.func1()
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:2482
created by golang.org/x/net/http2.bodyWriterState.scheduleBodyWrite
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:2533 +0xa1

goroutine 659315 [select]:
net/http/httputil.(*ReverseProxy).ServeHTTP.func1(0xc0009ce3f0, 0xc001146b10, 0x2921cc0, 0xc0004674c0)
	/usr/local/go/src/net/http/httputil/reverseproxy.go:190 +0xb7
created by net/http/httputil.(*ReverseProxy).ServeHTTP
	/usr/local/go/src/net/http/httputil/reverseproxy.go:189 +0x1469

goroutine 659811 [chan receive]:
net/http.http2closeWaiter.Wait(...)
	/usr/local/go/src/net/http/h2_bundle.go:3314
net/http.(*http2responseWriter).CloseNotify.func1(0xc0011232c0, 0xc0011d2620)
	/usr/local/go/src/net/http/h2_bundle.go:6147 +0x34
created by net/http.(*http2responseWriter).CloseNotify
	/usr/local/go/src/net/http/h2_bundle.go:6146 +0xf8

goroutine 659268 [select]:
golang.org/x/net/http2.(*ClientConn).roundTrip(0xc000a52000, 0xc003151100, 0x0, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1084 +0x9a0
golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0005b9140, 0xc003151100, 0xc00035af00, 0x3f2b9e8, 0xc00035afc8, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:447 +0x172
golang.org/x/net/http2.(*Transport).RoundTrip(...)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:408
github.com/containous/traefik/v2/pkg/server.(*h2cTransportWrapper).RoundTrip(0xc000138ef0, 0xc003151100, 0xc00011a3f0, 0x3, 0xc00035b048)
	/go/src/github.com/containous/traefik/pkg/server/roundtripper.go:23 +0x60
net/http.(*Transport).roundTrip(0xc0001d3a40, 0xc003151100, 0x1f9fea0, 0x1, 0xc000cc2580)
	/usr/local/go/src/net/http/transport.go:485 +0xd87
net/http.(*Transport).RoundTrip(0xc0001d3a40, 0xc003151100, 0x24bf627, 0xf, 0xc000cc26b0)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x35
net/http/httputil.(*ReverseProxy).ServeHTTP(0xc0005d0320, 0x2908e80, 0xc0007958a0, 0xc003151000)
	/usr/local/go/src/net/http/httputil/reverseproxy.go:249 +0x2a4
github.com/containous/traefik/v2/pkg/middlewares/pipelining.(*pipelining).ServeHTTP(0xc000497c50, 0x2908e80, 0xc0007958a0, 0xc003151000)
	/go/src/github.com/containous/traefik/pkg/middlewares/pipelining/pipelining.go:34 +0x6f
github.com/containous/traefik/v2/pkg/middlewares/accesslog.AddServiceFields(0x2908e80, 0xc0007958a0, 0xc003151000, 0x28c21a0, 0xc000497c50, 0xc000795820)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:47 +0x134
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc00013d8c0, 0x2908e80, 0xc0007958a0, 0xc003151000)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0x10b
github.com/vulcand/oxy/roundrobin.(*RoundRobin).ServeHTTP(0xc0004884d0, 0x2908e80, 0xc0007958a0, 0xc003150f00)
	/go/pkg/mod/github.com/vulcand/oxy@v1.0.0/roundrobin/rr.go:147 +0xfe
github.com/containous/traefik/v2/pkg/middlewares/emptybackendhandler.(*emptyBackend).ServeHTTP(0xc000497c90, 0x2908e80, 0xc0007958a0, 0xc003150f00)
	/go/src/github.com/containous/traefik/pkg/middlewares/emptybackendhandler/empty_backend_handler.go:31 +0x168
github.com/containous/traefik/v2/pkg/middlewares/tracing.(*forwarderMiddleware).ServeHTTP(0xc0004becc0, 0x2908e80, 0xc0007958a0, 0xc003150f00)
	/go/src/github.com/containous/traefik/pkg/middlewares/tracing/forwarder.go:38 +0x610
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc00013d900, 0x2908e80, 0xc0007958a0, 0xc003150f00)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:38 +0x144
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000194a80, 0x2908e80, 0xc0007958a0, 0xc003150f00)
	/go/pkg/mod/github.com/containous/mux@v0.0.0-20181024131434-c33f32e26898/mux.go:133 +0x104
github.com/containous/traefik/v2/pkg/middlewares/recovery.(*recovery).ServeHTTP(0xc00063c400, 0x2908e80, 0xc0007958a0, 0xc003150d00)
	/go/src/github.com/containous/traefik/pkg/middlewares/recovery/recovery.go:33 +0x131
github.com/containous/traefik/v2/pkg/middlewares/accesslog.AddOriginFields(0x2908e80, 0xc000795880, 0xc003150d00, 0x28c21e0, 0xc00063c400, 0xc000795820)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:55 +0x119
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc0001a2dc0, 0x2908e80, 0xc000795880, 0xc003150d00)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0x10b
github.com/containous/traefik/v2/pkg/middlewares/requestdecorator.(*RequestDecorator).ServeHTTP(0xc000138ef8, 0x2908e80, 0xc000795880, 0xc003150b00, 0xc000b0c9e8)
	/go/src/github.com/containous/traefik/pkg/middlewares/requestdecorator/request_decorator.go:47 +0x37d
github.com/containous/traefik/v2/pkg/middlewares/requestdecorator.WrapHandler.func1.1(0x2908e80, 0xc000795880, 0xc003150b00)
	/go/src/github.com/containous/traefik/pkg/middlewares/requestdecorator/request_decorator.go:84 +0x83
net/http.HandlerFunc.ServeHTTP(0xc00063c4a0, 0x2908e80, 0xc000795880, 0xc003150b00)
	/usr/local/go/src/net/http/server.go:2007 +0x44
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*Handler).ServeHTTP(0xc0005b9260, 0x2918540, 0xc00019a448, 0xc002147300, 0x28c84c0, 0xc00063c4a0)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:202 +0x9e0
github.com/containous/traefik/v2/pkg/middlewares/accesslog.WrapHandler.func1.1(0x2918540, 0xc00019a448, 0xc002147300)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:69 +0x5a
net/http.HandlerFunc.ServeHTTP(0xc00063c4c0, 0x2918540, 0xc00019a448, 0xc002147300)
	/usr/local/go/src/net/http/server.go:2007 +0x44
github.com/containous/traefik/v2/pkg/middlewares.(*HTTPHandlerSwitcher).ServeHTTP(0xc000138ee8, 0x2918540, 0xc00019a448, 0xc002147300)
	/go/src/github.com/containous/traefik/pkg/middlewares/handler_switcher.go:23 +0x70
github.com/containous/traefik/v2/pkg/middlewares/forwardedheaders.(*XForwarded).ServeHTTP(0xc00013f950, 0x2918540, 0xc00019a448, 0xc002147300)
	/go/src/github.com/containous/traefik/pkg/middlewares/forwardedheaders/forwarded_header.go:174 +0x10b
net/http.serverHandler.ServeHTTP(0xc00048e0e0, 0x2918540, 0xc00019a448, 0xc002147300)
	/usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.initNPNRequest.ServeHTTP(0x2921d80, 0xc00048c450, 0xc000593880, 0xc00048e0e0, 0x2918540, 0xc00019a448, 0xc002147300)
	/usr/local/go/src/net/http/server.go:3365 +0x8d
net/http.(*http2serverConn).runHandler(0xc000595980, 0xc00019a448, 0xc002147300, 0xc000893840)
	/usr/local/go/src/net/http/h2_bundle.go:5713 +0x9f
created by net/http.(*http2serverConn).processHeaders
	/usr/local/go/src/net/http/h2_bundle.go:5447 +0x4eb

goroutine 658878 [select]:
net/http/httputil.(*ReverseProxy).ServeHTTP.func1(0xc00026e1c0, 0xc00070c1f0, 0x2921cc0, 0xc000b40080)
	/usr/local/go/src/net/http/httputil/reverseproxy.go:190 +0xb7
created by net/http/httputil.(*ReverseProxy).ServeHTTP
	/usr/local/go/src/net/http/httputil/reverseproxy.go:189 +0x1469

goroutine 659779 [runnable]:
net/http.cloneURL(...)
	/usr/local/go/src/net/http/clone.go:26
net/http.(*Request).WithContext(...)
	/usr/local/go/src/net/http/request.go:360
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*Handler).ServeHTTP(0xc0005b9260, 0x2918540, 0xc0005bc1b0, 0xc001566300, 0x28c84c0, 0xc00063c4a0)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:167 +0x100c
github.com/containous/traefik/v2/pkg/middlewares/accesslog.WrapHandler.func1.1(0x2918540, 0xc0005bc1b0, 0xc001566300)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:69 +0x5a
net/http.HandlerFunc.ServeHTTP(0xc00063c4c0, 0x2918540, 0xc0005bc1b0, 0xc001566300)
	/usr/local/go/src/net/http/server.go:2007 +0x44
github.com/containous/traefik/v2/pkg/middlewares.(*HTTPHandlerSwitcher).ServeHTTP(0xc000138ee8, 0x2918540, 0xc0005bc1b0, 0xc001566300)
	/go/src/github.com/containous/traefik/pkg/middlewares/handler_switcher.go:23 +0x70
github.com/containous/traefik/v2/pkg/middlewares/forwardedheaders.(*XForwarded).ServeHTTP(0xc00013f950, 0x2918540, 0xc0005bc1b0, 0xc001566300)
	/go/src/github.com/containous/traefik/pkg/middlewares/forwardedheaders/forwarded_header.go:174 +0x10b
net/http.serverHandler.ServeHTTP(0xc00048e0e0, 0x2918540, 0xc0005bc1b0, 0xc001566300)
	/usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.initNPNRequest.ServeHTTP(0x2921d80, 0xc000c7e840, 0xc00057f500, 0xc00048e0e0, 0x2918540, 0xc0005bc1b0, 0xc001566300)
	/usr/local/go/src/net/http/server.go:3365 +0x8d
net/http.(*http2serverConn).runHandler(0xc000f05b00, 0xc0005bc1b0, 0xc001566300, 0xc000726460)
	/usr/local/go/src/net/http/h2_bundle.go:5713 +0x9f
created by net/http.(*http2serverConn).processHeaders
	/usr/local/go/src/net/http/h2_bundle.go:5447 +0x4eb

goroutine 659799 [runnable]:
golang.org/x/net/http2.(*ClientConn).roundTrip(0xc0007e4180, 0xc000cf1400, 0x0, 0x0, 0x0, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1084 +0x9a0
golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0005b9140, 0xc000cf1400, 0xc00035af00, 0x3f2b9e8, 0xc00035afc8, 0x0)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:447 +0x172
golang.org/x/net/http2.(*Transport).RoundTrip(...)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:408
github.com/containous/traefik/v2/pkg/server.(*h2cTransportWrapper).RoundTrip(0xc000138ef0, 0xc000cf1400, 0xc00011a4e0, 0x3, 0xc00035b048)
	/go/src/github.com/containous/traefik/pkg/server/roundtripper.go:23 +0x60
net/http.(*Transport).roundTrip(0xc0001d3a40, 0xc000cf1400, 0x1f9fea0, 0x1, 0xc0037a3ce0)
	/usr/local/go/src/net/http/transport.go:485 +0xd87
net/http.(*Transport).RoundTrip(0xc0001d3a40, 0xc000cf1400, 0x24bf627, 0xf, 0xc0037a3d98)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x35
net/http/httputil.(*ReverseProxy).ServeHTTP(0xc0005d0410, 0x2908e80, 0xc00044cd00, 0xc000cf1300)
	/usr/local/go/src/net/http/httputil/reverseproxy.go:249 +0x2a4
github.com/containous/traefik/v2/pkg/middlewares/pipelining.(*pipelining).ServeHTTP(0xc0001983b0, 0x2908e80, 0xc00044cd00, 0xc000cf1300)
	/go/src/github.com/containous/traefik/pkg/middlewares/pipelining/pipelining.go:34 +0x6f
github.com/containous/traefik/v2/pkg/middlewares/accesslog.AddServiceFields(0x2908e80, 0xc00044cd00, 0xc000cf1300, 0x28c21a0, 0xc0001983b0, 0xc00044cca0)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:47 +0x134
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc00013db00, 0x2908e80, 0xc00044cd00, 0xc000cf1300)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0x10b
github.com/vulcand/oxy/roundrobin.(*RoundRobin).ServeHTTP(0xc000488af0, 0x2908e80, 0xc00044cd00, 0xc000cf1200)
	/go/pkg/mod/github.com/vulcand/oxy@v1.0.0/roundrobin/rr.go:147 +0xfe
github.com/containous/traefik/v2/pkg/middlewares/emptybackendhandler.(*emptyBackend).ServeHTTP(0xc0001983f0, 0x2908e80, 0xc00044cd00, 0xc000cf1200)
	/go/src/github.com/containous/traefik/pkg/middlewares/emptybackendhandler/empty_backend_handler.go:31 +0x168
github.com/containous/traefik/v2/pkg/middlewares/tracing.(*forwarderMiddleware).ServeHTTP(0xc0004bf680, 0x2908e80, 0xc00044cd00, 0xc000cf1200)
	/go/src/github.com/containous/traefik/pkg/middlewares/tracing/forwarder.go:38 +0x610
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc00013dc80, 0x2908e80, 0xc00044cd00, 0xc000cf1200)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:38 +0x144
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000194a80, 0x2908e80, 0xc00044cd00, 0xc000cf1200)
	/go/pkg/mod/github.com/containous/mux@v0.0.0-20181024131434-c33f32e26898/mux.go:133 +0x104
github.com/containous/traefik/v2/pkg/middlewares/recovery.(*recovery).ServeHTTP(0xc00063c400, 0x2908e80, 0xc00044cd00, 0xc000cf0f00)
	/go/src/github.com/containous/traefik/pkg/middlewares/recovery/recovery.go:33 +0x131
github.com/containous/traefik/v2/pkg/middlewares/accesslog.AddOriginFields(0x2908e80, 0xc00044cce0, 0xc000cf0f00, 0x28c21e0, 0xc00063c400, 0xc00044cca0)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:55 +0x119
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*FieldHandler).ServeHTTP(0xc0001a2dc0, 0x2908e80, 0xc00044cce0, 0xc000cf0f00)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/field_middleware.go:36 +0x10b
github.com/containous/traefik/v2/pkg/middlewares/requestdecorator.(*RequestDecorator).ServeHTTP(0xc000138ef8, 0x2908e80, 0xc00044cce0, 0xc000cf0e00, 0xc000b0d9e8)
	/go/src/github.com/containous/traefik/pkg/middlewares/requestdecorator/request_decorator.go:47 +0x37d
github.com/containous/traefik/v2/pkg/middlewares/requestdecorator.WrapHandler.func1.1(0x2908e80, 0xc00044cce0, 0xc000cf0e00)
	/go/src/github.com/containous/traefik/pkg/middlewares/requestdecorator/request_decorator.go:84 +0x83
net/http.HandlerFunc.ServeHTTP(0xc00063c4a0, 0x2908e80, 0xc00044cce0, 0xc000cf0e00)
	/usr/local/go/src/net/http/server.go:2007 +0x44
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*Handler).ServeHTTP(0xc0005b9260, 0x2918540, 0xc00000e278, 0xc00448df00, 0x28c84c0, 0xc00063c4a0)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:202 +0x9e0
github.com/containous/traefik/v2/pkg/middlewares/accesslog.WrapHandler.func1.1(0x2918540, 0xc00000e278, 0xc00448df00)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:69 +0x5a
net/http.HandlerFunc.ServeHTTP(0xc00063c4c0, 0x2918540, 0xc00000e278, 0xc00448df00)
	/usr/local/go/src/net/http/server.go:2007 +0x44
github.com/containous/traefik/v2/pkg/middlewares.(*HTTPHandlerSwitcher).ServeHTTP(0xc000138ee8, 0x2918540, 0xc00000e278, 0xc00448df00)
	/go/src/github.com/containous/traefik/pkg/middlewares/handler_switcher.go:23 +0x70
github.com/containous/traefik/v2/pkg/middlewares/forwardedheaders.(*XForwarded).ServeHTTP(0xc00013f950, 0x2918540, 0xc00000e278, 0xc00448df00)
	/go/src/github.com/containous/traefik/pkg/middlewares/forwardedheaders/forwarded_header.go:174 +0x10b
net/http.serverHandler.ServeHTTP(0xc00048e0e0, 0x2918540, 0xc00000e278, 0xc00448df00)
	/usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.initNPNRequest.ServeHTTP(0x2921d80, 0xc000c7e840, 0xc00057f500, 0xc00048e0e0, 0x2918540, 0xc00000e278, 0xc00448df00)
	/usr/local/go/src/net/http/server.go:3365 +0x8d
net/http.(*http2serverConn).runHandler(0xc000f05b00, 0xc00000e278, 0xc00448df00, 0xc000717160)
	/usr/local/go/src/net/http/h2_bundle.go:5713 +0x9f
created by net/http.(*http2serverConn).processHeaders
	/usr/local/go/src/net/http/h2_bundle.go:5447 +0x4eb

goroutine 659818 [chan receive]:
net/http.http2closeWaiter.Wait(...)
	/usr/local/go/src/net/http/h2_bundle.go:3314
net/http.(*http2responseWriter).CloseNotify.func1(0xc000866600, 0xc0011d2690)
	/usr/local/go/src/net/http/h2_bundle.go:6147 +0x34
created by net/http.(*http2responseWriter).CloseNotify
	/usr/local/go/src/net/http/h2_bundle.go:6146 +0xf8

goroutine 659314 [chan receive]:
net/http.http2closeWaiter.Wait(...)
	/usr/local/go/src/net/http/h2_bundle.go:3314
net/http.(*http2responseWriter).CloseNotify.func1(0xc001122360, 0xc0009ce3f0)
	/usr/local/go/src/net/http/h2_bundle.go:6147 +0x34
created by net/http.(*http2responseWriter).CloseNotify
	/usr/local/go/src/net/http/h2_bundle.go:6146 +0xf8

goroutine 659819 [runnable]:
net/http/httputil.(*ReverseProxy).ServeHTTP.func1(0xc0011d2690, 0xc00087b790, 0x2921cc0, 0xc001340280)
	/usr/local/go/src/net/http/httputil/reverseproxy.go:189
created by net/http/httputil.(*ReverseProxy).ServeHTTP
	/usr/local/go/src/net/http/httputil/reverseproxy.go:189 +0x1469

goroutine 658768 [runnable]:
golang.org/x/net/http2.(*clientStream).awaitRequestCancel(0xc000df9180, 0xc000cf0d00)
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:312
created by golang.org/x/net/http2.(*clientConnReadLoop).handleResponse
	/go/pkg/mod/golang.org/x/net@v0.0.0-20190930134127-c5a3c61f89f3/http2/transport.go:1929 +0x6a8

goroutine 659812 [select]:
net/http/httputil.(*ReverseProxy).ServeHTTP.func1(0xc0011d2620, 0xc00087b360, 0x2921cc0, 0xc001340180)
	/usr/local/go/src/net/http/httputil/reverseproxy.go:190 +0xb7
created by net/http/httputil.(*ReverseProxy).ServeHTTP
	/usr/local/go/src/net/http/httputil/reverseproxy.go:189 +0x1469

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 15 (5 by maintainers)

Commits related to this issue

Most upvoted comments

Hello again @mpl, It looks like your changes introduced a new crash that seems to be happing on every request. I’ve attached the dump below.

I’ve create a quick and dirt setup for reproducing the original race condition: https://github.com/Monofraps/traefik-5825-repro Just run docker-compose up and wait a couple of minutes. Longest time it took me to hit the race was 4 minutes.

Crash from your changes:

time="2019-11-22T09:27:39Z" level=info msg="Configuration loaded from file: /var/traefik/traefik.toml"
panic: Header called after Handler finished

goroutine 54 [running]:
net/http.(*http2responseWriter).Header(0xc000138008, 0xc0002a3f80)
	/usr/local/go/src/net/http/h2_bundle.go:6158 +0x99
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*captureResponseWriter).Header(...)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/capture_response_writer.go:28
github.com/containous/traefik/v2/pkg/middlewares/accesslog.(*Handler).logTheRoundTrip(0xc00044e3c0, 0xc0002eb340, 0xc0002eb360, 0xc0004c2140)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:309 +0x8b3
github.com/containous/traefik/v2/pkg/middlewares/accesslog.NewHandler.func1(0xc00044e3c0)
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:125 +0xc2
created by github.com/containous/traefik/v2/pkg/middlewares/accesslog.NewHandler
	/go/src/github.com/containous/traefik/pkg/middlewares/accesslog/logger.go:122 +0x265
+2
nristock on Nov 22, 2019

@mpl Your fix definitely works. We haven’t seen a single crash since we’ve deployed the patch.

+1
nristock on Nov 28, 2019

@mpl Great work! Your fixed seems to be working. The hosts that had been patched this morning haven’t seen a crash since. I’ll report back tomorrow after our smoke tests ran but I’m 99% sure that the initial race has been fixed.

+1
nristock on Nov 27, 2019

@Monofraps hey. I’ve updated #5840 with a new commit that seems to pass your repro. Could you please try it out and let me know if it works for you now?

+1
mpl on Nov 26, 2019

Hi @mpl,

thanks for looking into it. I’m unable to provide a setup for reproduction right now. However, our setup is rather straight forward as we’re using traefik as an edge proxy and SSL/mTLS terminator for a couple of gRPC services only. We’re seeing around 20-30 crashes a day at ~80 requests per second with peaks up to a few hundred.

I’ll patch one of our nodes with the fix you provided and report back. If I have time, I’ll also try and create a minimal setup for reproducing the issue for you.

+1
nristock on Nov 22, 2019
Read more comments on GitHub
← traefik: Failing health check but still available
traefik: File Watch Doesn't Work →