caddy: http/2 transport failing for gRPC

caddy2 is currently not handling the http/2 protocol well. and this results that gRPC responses are not working.

the issue is that caddy is expecting a regular http1 response, but a http/2 stream response is not handled at all.

caddy should handle every http/2-request and http/2-responses as a unique http2-stream and proxy all frames to the same server and/or client until STREAM_END is received.

I made a simple demo project to reproduce the behavior: https://github.com/Zetanova/caddy-grpc-demo

Log output:

....
caddy_1           | {"level":"info","ts":1586265748.2906666,"logger":"http.log.access.log0","msg":"handled request","request":{"method":"POST","uri":"/
helloworld.Greeter/SayHello","proto":"HTTP/2.0","remote_addr":"172.24.0.1:47020","host":"localhost:50051","headers":{"Grpc-Timeout":["999985u"],"Conten
t-Type":["application/grpc"],"User-Agent":["grpc-go/1.29.0-dev"],"Te":["trailers"]}},"common_log":"172.24.0.1 - - [07/Apr/2020:13:22:28 +0000] \"POST /
helloworld.Greeter/SayHello HTTP/2.0\" 200 18","latency":0.0046437,"size":18,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["applicati
on/grpc"],"Trailer:Grpc-Status":["0"],"Trailer:Grpc-Message":[""]}}
greeter_client_1  | 2020/04/07 13:22:28 could not greet: rpc error: code = Internal desc = server closed the stream without sending trailers
greeter_client_1  | exit status 1

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 23 (15 by maintainers)

Commits related to this issue

caddyhttp: Fix trailers when recording responses (fixes #3236) — committed to caddyserver/caddy by mholt 4 years ago
caddyhttp: Fix trailers when recording responses (fixes #3236) — committed to caddyserver/caddy by mholt 4 years ago
httpserver: Add experimental H2C support (#3289) * reverse_proxy: Initial attempt at H2C transport/client support (#3218) I have not tested this yet * Experimentally enabling H2C server support... — committed to caddyserver/caddy by mholt 4 years ago

Most upvoted comments

@mholt It is working now with logging and the original golang gRPC sample client/server

Zetanova on Apr 21, 2020

@Zetanova I am pretty sure I have fixed it in 3c55cf9 in the h2c branch. Could you pull that and try it out? 😃

As a bonus, this should slightly improve performance.

mholt on Apr 21, 2020

@mholt The h2, h2c (tcp, uds) + gRPC support looks ok.

Zetanova on Apr 21, 2020

@mholt sry for the delay The error is persistent/same

your requested command:

$ grpcurl -d '{"name": "world"}' -plaintext host.docker.internal:50051 helloworld.Greeter/SayHello

Error invoking method "helloworld.Greeter/SayHello": failed to query for service descriptor "helloworld.Greeter": server does not support the reflection API

Server-Log:

caddy_1           | {"level":"info","ts":1587373186.2891846,"logger":"http.log.access.log0","msg":"handled request","req
uest":{"method":"POST","uri":"/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo","proto":"HTTP/2.0","remote
_addr":"172.24.0.1:32788","host":"host.docker.internal:50051","headers":{"Content-Type":["application/grpc"],"User-Agent
":["grpc-go/1.30.0-dev"],"Te":["trailers"]}},"common_log":"172.24.0.1 - - [20/Apr/2020:08:59:46 +0000] \"POST /grpc.refl
ection.v1alpha.ServerReflection/ServerReflectionInfo HTTP/2.0\" 200 0","latency":0.0026398,"size":0,"status":200,"resp_h
eaders":{"Server":["Caddy"],"Content-Type":["application/grpc"],"Grpc-Status":["12"],"Grpc-Message":["unknown service gr
pc.reflection.v1alpha.ServerReflection"]}}

modified command with proto file:

$ grpcurl -d '{"name": "world"}' -proto "helloworld/helloworld.proto" -plaintext host.docker.internal:50051 helloworld.Greeter/SayHello

ERROR:
  Code: Internal
  Message: server closed the stream without sending trailers

Server-Log:

greeter_server_1  | 2020/04/20 09:10:36 Received: world
caddy_1           | {"level":"info","ts":1587373836.3367207,"logger":"http.log.access.log0","msg":"handled request","req
uest":{"method":"POST","uri":"/helloworld.Greeter/SayHello","proto":"HTTP/2.0","remote_addr":"172.24.0.1:32836","host":"
host.docker.internal:50051","headers":{"Content-Type":["application/grpc"],"User-Agent":["grpc-go/1.30.0-dev"],"Te":["tr
ailers"]}},"common_log":"172.24.0.1 - - [20/Apr/2020:09:10:36 +0000] \"POST /helloworld.Greeter/SayHello HTTP/2.0\" 200
18","latency":0.0037791,"size":18,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["application/grpc"],"T
railer:Grpc-Message":[""],"Trailer:Grpc-Status":["0"]}}
caddy_1           | {"level":"info","ts":1587373836.3367207,"logger":"http.log.access.log0","msg":"handled request","req
uest":{"method":"POST","uri":"/helloworld.Greeter/SayHello","proto":"HTTP/2.0","remote_addr":"172.24.0.1:32836","host":"
host.docker.internal:50051","headers":{"Content-Type":["application/grpc"],"User-Agent":["grpc-go/1.30.0-dev"],"Te":["tr
ailers"]}},"common_log":"172.24.0.1 - - [20/Apr/2020:09:10:36 +0000] \"POST /helloworld.Greeter/SayHello HTTP/2.0\" 200
18","latency":0.0037791,"size":18,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["application/grpc"],"T
railer:Grpc-Message":[""],"Trailer:Grpc-Status":["0"]}}

Zetanova on Apr 20, 2020