caddy: caddy+QUIC crash when serving static content

1. What version of Caddy are you running (`caddy -version`)?

Compiled from source, (untracked dev build).

2. What are you trying to do?

Serve web pages with QUIC over lossy and slow network connections.

3. What is your entire Caddyfile?

$hostname:443
root $webRoot
gzip
log $logFolder/caddy.access.log
errors $logFolder/caddy.error.log

tls $certFolder/cert.pem $certFolder/key.pem {
	protocols tls1.2 tls1.2
}

4. How did you run Caddy (give the full command and describe the execution environment)?

caddy -quic -conf=“a_folder/config/netem.Caddyfile” -pidfile=“a_pidfile”

5. What did you expect to see?

Caddy serving web pages without crashing, as it does with the same settings but using HTTP/1.1 or HTTP/2.

6. What did you see instead (give full error messages and/or log)?

goroutine 2716 [running]:
panic(0x9cd760, 0xc4200120f0)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/lucas-clemente/quic-go.(*stream).AddStreamFrame(0x0, 0xc420edd260, 0x0, 0x0)
	/home/fille/go/src/github.com/lucas-clemente/quic-go/stream.go:238 +0x48
github.com/lucas-clemente/quic-go.(*stream).CloseRemote(0x0, 0x0)
	/home/fille/go/src/github.com/lucas-clemente/quic-go/stream.go:255 +0x79
github.com/lucas-clemente/quic-go/h2quic.(*Server).handleRequest(0xc420225720, 0xd28660, 0xc4209bb860, 0xd2d5c0, 0xc4202ba120, 0xc4206f2f50, 0xc420068270, 0xc420068340, 0x0, 0x0)
	/home/fille/go/src/github.com/lucas-clemente/quic-go/h2quic/server.go:160 +0x762
github.com/lucas-clemente/quic-go/h2quic.(*Server).handleStream.func1(0xc420225720, 0xd28660, 0xc4209bb860, 0xd2d5c0, 0xc4202ba120, 0xc420068270, 0xc420068340)
	/home/fille/go/src/github.com/lucas-clemente/quic-go/h2quic/server.go:108 +0xbf
created by github.com/lucas-clemente/quic-go/h2quic.(*Server).handleStream
	/home/fille/go/src/github.com/lucas-clemente/quic-go/h2quic/server.go:119 +0x116

7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?

Load web pages with QUIC continuously over bad network links (happens on many different web pages). I’m using traffic control within Linux in order to create different network conditions.

I’ve created a simple proof-of-concept which can spawn a shell with poor network conditions. The problem with caddy is something that possibly only occurs when the client resides on poor networks, but I’ll need to figure this out by testing. The tool I’ve created is only tested on Ubuntu 16.10: https://github.com/hansfilipelo/netem

I’ll attach a list of my test domains that I’ve scraped and are serving as static content with Caddy. urls.txt

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 33 (15 by maintainers)

Most upvoted comments

@hansfilipelo I was about to write the same about my comment - reading it now it sounds angry to me as well 😉 Turns out I didn’t read your steps carefully enough, so of course I couldn’t reproduce the issue. Sorry for that!

I’m now loading the wgetted version of login.tmall.com via QUIC. So thanks for the detailed steps.

marten-seemann on Feb 15, 2017

@hansfilipelo Thanks for the log file, that was really helpful.

Can you please rebuild Caddy (don’t forget to get the QUIC update first) and try again? I’m pretty certain the bug is fixed now, but I’d like to be 100% sure.

marten-seemann on Feb 10, 2017

I am unable to replicate the issue now. Great work @marten-seemann.

Fixed with commit c7b4ad6e in quic-go (https://github.com/lucas-clemente/quic-go/commit/c7b4ad6e80bcc617ad58db5486c279eafa74415a).

hansfilipelo on Feb 16, 2017

@marten-seemann This is a minimal example.

Can’t really fetch this site with Caddy over QUIC without downloading it first. I’m not the developer of that site and have no affiliation with tmall.com and I do not even know what company tmall is - hence I have no idea what they are doing to trigger the caddy crash.
You can serve the downloaded site on any domain, including your own that has a valid certificate (which you will need since you are running QUIC in a Chromium based browser) so you do not need to edit your hosts file. This is thanks to wget rewriting links on the page you download to relative links.
See 2.

Caddyfile is posted in the first post of the issue but here it is again with comments:

$hostname:443 # Here you put your hostname
root $webRoot # This is where you'll store the index of the site you've fetched with wget
gzip
log $logFolder/caddy.access.log
errors $logFolder/caddy.error.log # Log lines probably does not matter

tls $certFolder/cert.pem $certFolder/key.pem { # These could be generated automatically if DNS is set globally
	protocols tls1.2 tls1.2
}

hansfilipelo on Feb 15, 2017