caddy: Some proxied POST requests fail or hang when upstream does not consume body

1. What version of Caddy are you running (caddy -version)?

Caddy (untracked dev build)

2. What are you trying to do?

Upload files to a reverse proxy

3. What is your entire Caddyfile?

site.com {
  proxy / 127.0.0.1:1333 {
    max_fails 0
  }
}

4. How did you run Caddy (give the full command and describe the execution environment)?

$ sudo caddy -email="user@host.com" -agree

5. What did you expect to see?

Proxied requests and responses

6. What did you see instead (give full error messages and/or log)?

Requests were either not sent to the proxy server and responses were only partially sent (just the headers). Usually requests either end up hanging on the client side and never reach the proxy or the responses return the headers with a protocol exception (net::ERR_SPDY_PROTOCOL_ERROR)

This occurs when the proxy returns a status code outside the range of 200-399 despite having max_fails 0 set to prevent the backend from being marked as down. Even if this were the case, then this would not be expected behavior

No error messages are reported by Caddy when adding -log stdout to the command or anywhere else.

ss 2016-04-22 at 08 43 13

Code to reproduce (use the network inspector to view details)

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 1
  • Comments: 39 (17 by maintainers)

Commits related to this issue

Most upvoted comments

I believe one half of this issue is caused by a bug in golang/go#15425. Note in my bug report, I can reproduce the issue without even using caddy just the original script @6f7262 provided.

However, this really only explains why it doesn’t work on http/2, the other failures are interesting…

As in, retry a max of 4 times in 60 seconds or something.

That’s the equivalent of setting max_fails to 4. When max_fails is 0, it means to mark the backend as never down, and as long as there exists non-down backends, Caddy will keep trying for 60s.

In fact the “fix” I suggested is “equivalent” (it’s racy) to

max_fails 1
fail_duration 1ms

Whoops. Misclick.

I agree, all of the behaviors in this are bizarre. I also noticed that it’s not even about proxying anymore, but rather just about Go’s responses.

That’s what I thought too, but I still see similar behavior with http2 disabled (run caddy with -http2=false).

Looks like a bug in golang’s http2 library.

My guess is if in a http2 server, a request’s Body is closed before it is fully read, for some reason the responsewriter’s body is discarded.