caddy: Caddy Crashing Due To Nil Pointer Dereference
On a production server, getting the following error:
an 31 00:24:41 Dw-Haproxy caddy[2568599]: panic: runtime error: invalid memory address or nil pointer dereference
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x5651f2]
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: goroutine 907455 [running]:
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: bufio.(*Writer).Available(...)
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: bufio/bufio.go:646
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: bufio.(*Writer).Write(0x0, {0x2a11ac0?, 0xc006e4b880?, 0xc00a8516ab?})
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: bufio/bufio.go:664 +0x52
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: net/http.(*response).WriteHeader(0xc00a8515e0, 0x64)
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: net/http/server.go:1165 +0x20e
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).reverseProxy.func1(0xc00490ee40?, 0xc008249b00?)
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: github.com/caddyserver/caddy/v2@v2.6.2/modules/caddyhttp/reverseproxy/reverseproxy.go:783 +0x6f
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: net/http.(*persistConn).readResponse(0xc00a0a6120, {{}, 0xc008249b00, {0xc008249a00}, 0xc0096cd020, 0x0, 0xc0096ccf60, 0xc0096ccfc0}, 0xc008155880)
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: net/http/transport.go:2313 +0x20c
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: net/http.(*persistConn).readLoop(0xc00a0a6120)
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: net/http/transport.go:2108 +0x3aa
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: created by net/http.(*Transport).dialConn
Jan 31 00:24:41 Dw-Haproxy caddy[2568599]: net/http/transport.go:1751 +0x173e
Jan 31 00:24:41 Dw-Haproxy systemd[1]: caddy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 31 00:24:41 Dw-Haproxy systemd[1]: caddy.service: Failed with result 'exit-code'.
The config being used is:
sub.domain.tld {
reverse_proxy localhost:8080 {
#header_up X-Forwarded-For {header.Cf-Connecting-Ip}
}
route /stats {
reverse_proxy localhost:2087
}
handle_path /hidden/* {
reverse_proxy 0.0.0.0:3000
}
}
sub2.domain.tld {
reverse_proxy localhost:8080
route /stats {
reverse_proxy localhost:2087
}
}
The reverse_proxy 0.0.0.0:3000 part uses a different ip, and was replaced for 0.0.0.0 while sharing to hide sensitive info.
This has worked without issues for multiple months until today when crashes started. A simple restart fixes the issue and gets the application working.
➜ ~ caddy version
v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 15 (11 by maintainers)
@dunglas I created a patch for your plugin. That issues doesn’t have anything to with the error this issue described.
From the log it appears to be a bug with stdlib. Maybe a corner case in net/http/transport
I’ve enabled request logging and once I see another crash, I’ll pass the logs back here.
I assume this is working alright now with the patch upstream, so I’ll close this. Thanks everyone!
Just an update from my side, it hadn’t crashed again since I’ve enabled request logging. Still waiting to see if it will though.
Thank you very much, and sorry for the false report.
I remember once I also encountered this bug, that’s about at least 2 years before. Way before 103 is introduced.
Bizarre:
https://cs.opensource.google/go/go/+/refs/tags/go1.19.5:src/net/http/server.go;l=1165
and
https://cs.opensource.google/go/go/+/refs/tags/go1.19.5:src/bufio/bufio.go;l=645;drc=007d8f4db1f890f0d34018bb418bdc90ad4a8c35
Like the buffer of the connection is nil? How is that even possible?
Maybe it’s a bug in our code, but I wonder if there could be a very unlikely bug in Go std lib.