caddy: Fatal error: concurrent map read and map write

1. What version of Caddy are you running (caddy -version)?

v0.9.1

2. What are you trying to do?

Refresh heavily a web site served by caddy reverse proxy, i.e. hold F5 key for a while in Chrome / Firefox.

3. What is your entire Caddyfile?

https://:443 {
 tls self_signed
 proxy / 127.0.0.1:580 {
    transparent
    max_fails 0
  }
  bind 127.0.0.1
}

4. How did you run Caddy (give the full command and describe the execution environment)?

caddy.exe -conf Caddyfile

5. What did you expect to see?

I expected the web site to appear once the heavy refreshing is done.

6. What did you see instead (give full error messages and/or log)?

The web site kept loading indefinitely (spinning wheel). Refresh or restart of browser did not help. Then Caddy failed with fatal error (see attachment for complete output):

Activating privacy features... done.
https:// (only accessible on this machine)
fatal error: concurrent map read and map write

goroutine 19217 [running]:
runtime.throw(0xa3a054, 0x21)
    /usr/local/go/src/runtime/panic.go:566 +0x9c fp=0xc0458ffc38 sp=0xc0458ffc18
runtime.mapaccess1_faststr(0x9ca040, 0xc04217d9b0, 0xa2aff8, 0xa, 0x94ce9aa0244a93f8)
    /usr/local/go/src/runtime/hashmap_fast.go:201 +0x4fa fp=0xc0458ffc98 sp=0xc0458ffc38
net/textproto.MIMEHeader.Get(0xc04217d9b0, 0xa2aff8, 0xa, 0xc04232eb00, 0x0)
    /usr/local/go/src/net/textproto/header.go:33 +0x7c fp=0xc0458ffcd0 sp=0xc0458ffc98
net/http.Header.Get(0xc04217d9b0, 0xa2aff8, 0xa, 0xa, 0xc04232eb88)
    /usr/local/go/src/net/http/header.go:39 +0x46 fp=0xc0458ffd08 sp=0xc0458ffcd0
net/http.(*Request).write(0xc042324d20, 0xcbbd20, 0xc04472c6c0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/request.go:507 +0xabf fp=0xc0458ffe48 sp=0xc0458ffd08
net/http.(*persistConn).writeLoop(0xc042c71700)
    /usr/local/go/src/net/http/transport.go:1644 +0x1b3 fp=0xc0458fff98 sp=0xc0458ffe48
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc0458fffa0 sp=0xc0458fff98
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1058 +0x515
...

7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?

Unfortunately, I could not reproduce the issue by proxying to another Caddy instance or a public Internet site. I can only reproduce the issue with our application as upstream host running on localhost. It seems to be dependent on timing and number of requests from browser.

The issue does not occur when running caddy with HTTP2 disabled, when running without proxy or when running with proxy but without max_fails directive. I also could not reproduce the issue with boom benchmark tool (tried w/ and w/o -h2 switch).

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 22 (11 by maintainers)

Most upvoted comments

@mholt I think it’s because we reuse the outreq when retrying, but the standard lib may still use it after RoundTrip. So we could make a new outreq before each iteration of retrying loop for now.

BTW, just quote from the doc of RoundTrip:

// RoundTrip must always close the body, including on errors, // but depending on the implementation may do so in a separate // goroutine even after RoundTrip returns. This means that // callers wanting to reuse the body for subsequent requests // must arrange to wait for the Close call before doing so.

So I think possibility of reuse may exists…

Also I think this issue only reproduced on http2 path and also forward the request to proxy.

Yeah, I will rebuild my container and if it happens again get a log out.

https://gist.github.com/anonymous/146fe3bcaa53505978954a412324f911

fatal error: concurrent map writes

I received this on Caddy 0.9.0 running within docker on Ubuntu 16.04. Self made container downloading caddy from caddyserver.com with zero additional features enabled.