caddy: cannot reverse proxy in front of grpc + grpc-web multiplexer

5 is THE issue. The problem is that ALL web browsers are still HTTP 1.1. Thus there’s a need in a BRIDGE, which converts HTTP1.1 to HTTP2. Reverse proxy is that bridge.

My system works with Envoy as a reverse proxy. It takes HTTP 1.1 input from a grpc-web and converts into native HTTP 2.0, which is grpc.

Remember, there’s no grpc-web protocol. Those are just two projects from here and here, which solve the above stated issue - convert HTTP 1.1 into HTTP2.0 transport

The proxy still needs to do the work to translate gRPC-Web bytes into gRPC bytes, which is the work done by Envoy and the gRPC-Web proxy provided by Improbable. This is what Johan says in his blog post. Currently, Caddy does not have this translation layer between the 2 protocols natively. It may be developed as a module.

That said, and this comes with a disclaimer of “I have not tested this myself”, you can have Caddy sit between the client and Envoy and/or between Envoy and the backend; but you cannot expect Caddy to translate gRPC-Web to gRPC. At least not now. Unless you want to turn this into a feature request, which probably should go into a new ticket for discussion.

@mohammed90 True, although I wonder if he means across different programs, i.e. not just Caddy. It’s true other programs can use those certs too, but they have to know to look for them there.

@mholt The caddy reverse proxy configuration is practically the same except I had set flush_interval to -1.

{
    "handler": "reverse_proxy",
    "flush_interval": -1,
                  "transport": {
                    "protocol": "http",
                    "versions": [ "1.1", "h2c"]
                  },
}

I think the main issue stems from the fact that grpc-web requests are actually incompatible with a grpc server because the protocols are slightly different. The general solution is to have a reverse proxy like Envoy sit between the client and the grpc server and I’m not sure Caddy supports translating the requests like Envoy. My solution for this was to use grpcweb to create a grpc-web compatible server and use a connection multiplexer to host both grpc-web and grpc on the same port. This is where I ran into issues, and I suspect this is also where @tiero had problems.

Caddy was only sending h2c requests to my webserver so while my grpc handler could understand it, my grpc-web handler could not and I was getting read: connection reset by peer errors. At first I thought this was a Caddy issue, but it looks to be an issue with my Go server. Since the default http server in Go does not handle h2c I had to modify the multiplexer to

1. Match any http2 request with application/grpc headers with the grpc server
2. Match any h2c request with application/grpc-web headers with a http2 server
3. Match any http1.1 request with application/grpc-web headers with the default http server

TLDR: This really an issue with Caddy but an issue with how we wrote our gRPC servers. Caddy does not translate grpc-web to grpc so our Go servers need to handle grpc-web requests. Additionally, we have to explicitly use the h2 package grpc-web requests since the default go http server does not serve h2 by default.

@mholt The reason to use mkcert is to allow TLS on the backend. I need all services to talk natively with mTLS to each other. I had a project with Istio, which has an automatic cert management in control plane with istio-csr.

For Caddy, there’s a lack of a higher level control plane which would allow sharing certificates between different services. That’s why I need to manually generate certs and manage them. Later in production, I may again turn to Service Mesh to avoid pains with the manual cert management 🙄

You can use a shared storage backend besides the default of filesystem to share the certs. For instance, there are two module to utilize Consul as the certificate storage, from which you can grab the certs to be shared across instances.

Yeah, all this is redundant and can be removed:

    transport http {
      versions h2c 2
      dial_timeout 3s
    }

The dial timeout as of v2.5.0 (iirc) is 3s, and using h2c:// sets those versions automatically

@mholt I’m not sure I understood your question. Here’s what I have

1. Go GRPC server without TLS
2. Go GRPC client without TLS
3. NodeJS GPRC client without TLS
4. grpcurl plaintext without TLS
5. TypeScript client for Web Browser, which uses Envoy and Caddy as a reverse proxy for grpc-web requests

2-4 can communicate with 1 without issues. Both TLS and plaintext. All three are using GRPC native transport, which is HTTP 2.0 based.

5 is THE issue. The problem is that ALL web browsers are still HTTP 1.1. Thus there’s a need in a BRIDGE, which converts HTTP1.1 to HTTP2. Reverse proxy is that bridge.

My system works with Envoy as a reverse proxy. It takes HTTP 1.1 input from a grpc-web and converts into native HTTP 2.0, which is grpc.

Remember, there’s no grpc-web protocol. Those are just two projects from here and here, which solve the above stated issue - convert HTTP 1.1 into HTTP2.0 transport

So, bottom line. How do I config Caddy to deliver that HTTP 1.1 to HTTP 2.0 transport bridging? Envoy does the job, here is an example

@jbrown-stripe might be a longshot, but could you try building from the latest from master?

I think there’s a chance https://github.com/caddyserver/caddy/commit/e6c29ce081673d85e527d59f3afb7ace034573df has fixed this issue, but I may not be fully understanding the problem. Just a sanity check to make sure this is still an issue.

Edit: You can try v2.4.5 at this point, which has the aforementioned fixes.

@tiero It looks like both grpc-web and grpc request get sent to the grpc listener which won’t work. Instead only grpc requests should be routed to the grpc listener and then grpc-web should be routed to http1.1 and h2c listener.

@tiero Running into the exact same issue, any luck figuring it out?

No unfortunately. I ended up implementing TLS via certmagic in gocode directly

But would be nice to use caddyserver anyway, bur cannot figure out still

@tiero I ended up getting this to work a few days later. I never needed http2 with tls, just h2c. Regardless, caddy is sending h2c requests when proxying grpc-web. However your http listener does not match it and cannot handle the request. It also doesn’t like h2c request can be handled by the default http server in go. My solution was to

1. set flush_interval=1 in caddy. This seemed to fix the stream errors I was running into.
2. Explicitly handle h2c and http 1.1 requests within go

cMux := cmux.New(lis)
	grpcListener := cMux.MatchWithWriters(cmux.HTTP2MatchHeaderFieldSendSettings("content-type", "application/grpc"))
	grpcWebHTTP2Listener := cMux.Match(cmux.HTTP2())
	grpcWebHTTP1Listener := cMux.Match(cmux.HTTP1())

	wrappedGrpc := grpcweb.WrapServer(s.grpc)
	handleGrpcWebReq := func(w http.ResponseWriter, req *http.Request) {
		if wrappedGrpc.IsGrpcWebRequest(req) {
			wrappedGrpc.ServeHttp(w, req)
			return
		}

		msg := "received a request that could not be matched to grpc or grpc-web"
		log.Error(msg)
		w.WriteHeader(http.StatusNotFound)
		w.Write([]byte(msg))
	}

	mux := http.NewServeMux()
	mux.HandleFunc("/", handleGrpcWebReq)

	httpSrv := &http.Server{
		Addr:    s.address,
		Handler: mux,
	}

	http2Srv := &http.Server{
		Addr:    s.address,
		Handler: h2c.NewHandler(http.HandlerFunc(handleGrpcWebReq), &http2.Server{}),
	}

	serveGRPC(s.grpc, grpcListener)
	serveHTTP(http2Srv, grpcWebHTTP2Listener)
	serveHTTP(httpSrv, grpcWebHTTP1Listener)

func serveHTTP(s *http.Server, l net.Listener) {
	go func() {
		if err := s.Serve(l); err != nil {
			log.Fatal("error starting http server", log.Err(err))
		}
	}()
}

func serveGRPC(s *grpc.Server, l net.Listener) {
	go func() {
		if err := s.Serve(l); err != nil {
			log.Fatal("error starting grpc server", log.Err(err))
		}
	}()
}

@tiero Running into the exact same issue, any luck figuring it out?