ingress-nginx: gRPC error: server closed the stream without sending trailers
NGINX Ingress controller version: v0.17.1 and v0.18.0
Kubernetes version (use kubectl version
):
v1.10.5 and v1.10.6
Environment:
- Cloud provider or hardware configuration: Azure AKS and GKE
- OS (e.g. from /etc/os-release): 16.04.2-Ubuntu
- Kernel (e.g.
uname -a
): 4.15.0-1013-azure - Install tools: helm
- Others: google.golang.org/grpc: 1.14.0 github.com/golang/protobuf: 1.2.0
What happened:
The grpc request was successfully proxied to the backend server (written in Go), which returned data back to nginx, but then the response to the client was rpc error: code = Internal desc = server closed the stream without sending trailers
.
If I use kubectl port-forward
and WithInsecure
, the gRPC request works perfectly.
198.217.127.184 - [198.217.127.184] - - [21/Aug/2018:05:09:58 +0000] "POST /keywordspb.KeywordsService/BatchGetKeywords HTTP/1.1" 200 120 "-" "/-2ef21c322cee29c3de4c2b0fe29f93b9faecb33d grpc-go/1.14.0" 512 0.069 [keywords-keywords-grpc] 10.0.2.211:8888 119 0.068 200 05c7736fee61324d03570abf6338efb0
What you expected to happen: nginx to terminate SSL and then forward the insecure gRPC request to the upstream server, then successfully return the response
How to reproduce it (as minimally and precisely as possible):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: keywords
name: keywords
labels:
app: keywords
annotations:
kubernetes.io/ingress.class: "nginx"
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/grpc-backend: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
tls:
- secretName: keywords-tls
hosts:
- keywords.company.app
rules:
- host: keywords.company.app
http:
paths:
- backend:
serviceName: keywords
servicePort: grpc
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 18 (15 by maintainers)
Ok, I feel like an idiot. The issue was that Cloudflare was intercepting the traffic and killing the http2 connection to nginx. Sorry for the noise and thanks for your help.
@derekperkins glad you found the issue, and it wasn’t related to the ingress controller.