go: net/http/httputil: ReverseProxy doesn't proxy grpc without help due to 'te' header handling

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

Go 1.8.3 (but can reproduce with 1.9beta2.

What operating system and processor architecture are you using (go env)?

darwin and linux

What did you do?

Tried to use the http2 transport to talk to a remote http2 server. https://play.golang.org/p/W_HbQGJZK9

The server on the other end is the grpc helloworld example. When running against the grpc-go implementation, the code works. When running agains ruby or python(both backed by C code) I get the INTERNAL_ERROR response described above. Logging on the server side itself is silent.

Note, the HTTP POST is only to more easily reproduce the issue. What I’m trying to do instead is use httputil.ReverseProxy to proxy grpc traffic: https://play.golang.org/p/micZul79Ow This succeeds if the upstream server is in Go

C server examples: https://github.com/grpc/grpc/tree/master/examples (connecting to these fails) grpc-go examples: https://github.com/grpc/grpc-go/tree/master/examples (connecting to this server succeeds)

What did you expect to see?

An HTTP response.

What did you see instead?

GODEBUG=http2debug=2 go run main.go                                                                                                                                                                    
2017/07/19 23:34:05 http2: Transport creating client conn 0xc42013c000 to [::1]:8082
2017/07/19 23:34:05 http2: Framer 0xc42014e000: wrote SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2017/07/19 23:34:05 http2: Framer 0xc42014e000: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824
2017/07/19 23:34:05 http2: Transport encoding header ":authority" = "localhost:8082"
2017/07/19 23:34:05 http2: Transport encoding header ":method" = "POST"
2017/07/19 23:34:05 http2: Transport encoding header ":path" = "/"
2017/07/19 23:34:05 http2: Transport encoding header ":scheme" = "https"
2017/07/19 23:34:05 http2: Transport encoding header "content-type" = "application/grpc"
2017/07/19 23:34:05 http2: Transport encoding header "content-length" = "5"
2017/07/19 23:34:05 http2: Transport encoding header "accept-encoding" = "gzip"
2017/07/19 23:34:05 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2017/07/19 23:34:05 http2: Framer 0xc42014e000: wrote HEADERS flags=END_HEADERS stream=1 len=51
2017/07/19 23:34:05 http2: Framer 0xc42014e000: wrote DATA stream=1 len=5 data="hello"
2017/07/19 23:34:05 http2: Framer 0xc42014e000: read SETTINGS len=24, settings: INITIAL_WINDOW_SIZE=65535, MAX_FRAME_SIZE=65538, MAX_HEADER_LIST_SIZE=16384, UNKNOWN_SETTING_65027=1
2017/07/19 23:34:05 http2: Framer 0xc42014e000: wrote DATA flags=END_STREAM stream=1 len=0 data=""
2017/07/19 23:34:05 http2: Transport received SETTINGS len=24, settings: INITIAL_WINDOW_SIZE=65535, MAX_FRAME_SIZE=65538, MAX_HEADER_LIST_SIZE=16384, UNKNOWN_SETTING_65027=1
2017/07/19 23:34:05 Unhandled Setting: [MAX_HEADER_LIST_SIZE = 16384]
2017/07/19 23:34:05 Unhandled Setting: [UNKNOWN_SETTING_65027 = 1]
2017/07/19 23:34:05 http2: Framer 0xc42014e000: wrote SETTINGS flags=ACK len=0
2017/07/19 23:34:05 http2: Framer 0xc42014e000: read SETTINGS flags=ACK len=0
2017/07/19 23:34:05 http2: Transport received SETTINGS flags=ACK len=0
2017/07/19 23:34:05 http2: Framer 0xc42014e000: read RST_STREAM stream=1 len=4 ErrCode=INTERNAL_ERROR
2017/07/19 23:34:05 http2: Transport received RST_STREAM stream=1 len=4 ErrCode=INTERNAL_ERROR
2017/07/19 23:34:05 http2: Framer 0xc42014e000: read HEADERS flags=END_STREAM|END_HEADERS stream=1 len=83
2017/07/19 23:34:05 http2: decoded hpack field header field ":status" = "200"
2017/07/19 23:34:05 http2: decoded hpack field header field "content-type" = "application/grpc"
2017/07/19 23:34:05 RoundTrip failure: stream error: stream ID 1; INTERNAL_ERROR
2017/07/19 23:34:05 http2: decoded hpack field header field "grpc-status" = "1"
2017/07/19 23:34:05 http2: decoded hpack field header field "grpc-message" = "Cancelled"
2017/07/19 23:34:05 http2: Transport received HEADERS flags=END_STREAM|END_HEADERS stream=1 len=83
2017/07/19 23:34:05 Post https://localhost:8082: stream error: stream ID 1; INTERNAL_ERROR

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 15 (11 by maintainers)

Commits related to this issue

Most upvoted comments

I have the same problem but cant find a solution: I have a ProxyServer written in go

func main() {
	log.SetPrefix("[proxy] ")
	log.SetOutput(os.Stdout)
	if upstream == "" {
		log.Fatal("ERROR: missing argument upstream ")
	}
	url, _ := url.Parse(upstream)
	proxy := &Upstream{target: url, proxy: httputil.NewSingleHostReverseProxy(url)}
	mux := http.NewServeMux()
	mux.HandleFunc("/", proxy.handle)
	log.Fatal(http.ListenAndServeTLS(httpsSocket, httpsCert, httpsKey, mux))
}

type Upstream struct {
	target *url.URL
	proxy  *httputil.ReverseProxy
}

func (p *Upstream) handle(w http.ResponseWriter, r *http.Request) {
	p.proxy.ServeHTTP(w, r)
}

If I make a call from gnmi server to client, I get:

[proxy] 2019/01/16 12:34:57 http: proxy error: net/http: HTTP/1.x transport connection broken: write tcp 127.0.0.1:55595->127.0.0.1:10161: write: broken pipe
[proxy] 2019/01/16 12:35:15 http: proxy error: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x00\x04\x00\x00\x00\x00\x00"
[proxy] 2019/01/16 12:35:31 http: proxy error: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x00\x04\x00\x00\x00\x00\x00"

Any hints?
+1
idefixcert on Jan 16, 2019
Read more comments on GitHub
← go: net/http/httputil: Proxy terminates HTTP/2 stream before reading response body.
go: net/http/httputil: ReverseProxy no longer uses io.ReaderFrom/WriterTo →