ingress-nginx: gRPC error: server closed the stream without sending trailers

NGINX Ingress controller version: v0.17.1 and v0.18.0

Kubernetes version (use kubectl version): v1.10.5 and v1.10.6

Environment:

  • Cloud provider or hardware configuration: Azure AKS and GKE
  • OS (e.g. from /etc/os-release): 16.04.2-Ubuntu
  • Kernel (e.g. uname -a): 4.15.0-1013-azure
  • Install tools: helm
  • Others: google.golang.org/grpc: 1.14.0 github.com/golang/protobuf: 1.2.0

What happened: The grpc request was successfully proxied to the backend server (written in Go), which returned data back to nginx, but then the response to the client was rpc error: code = Internal desc = server closed the stream without sending trailers.

If I use kubectl port-forward and WithInsecure, the gRPC request works perfectly.

198.217.127.184 - [198.217.127.184] - - [21/Aug/2018:05:09:58 +0000] "POST /keywordspb.KeywordsService/BatchGetKeywords HTTP/1.1" 200 120 "-" "/-2ef21c322cee29c3de4c2b0fe29f93b9faecb33d grpc-go/1.14.0" 512 0.069 [keywords-keywords-grpc] 10.0.2.211:8888 119 0.068 200 05c7736fee61324d03570abf6338efb0

What you expected to happen: nginx to terminate SSL and then forward the insecure gRPC request to the upstream server, then successfully return the response

How to reproduce it (as minimally and precisely as possible):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: keywords
  name: keywords
  labels:
    app: keywords
  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/grpc-backend: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"

spec:
  tls:
  - secretName: keywords-tls
    hosts:
    - keywords.company.app
  rules:
  - host: keywords.company.app
    http:
      paths:
      - backend:
          serviceName: keywords
          servicePort: grpc

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 18 (15 by maintainers)

Most upvoted comments

Ok, I feel like an idiot. The issue was that Cloudflare was intercepting the traffic and killing the http2 connection to nginx. Sorry for the noise and thanks for your help.

@derekperkins glad you found the issue, and it wasn’t related to the ingress controller.