ingress-nginx: gRPC doesn't work

NGINX Version: nginx -v nginx/1.19.9 NGINX Installation: https://kubernetes.github.io/ingress-nginx/deploy/#azure Platform: Azure AKS

Problem Description

Hi Everyone, I tried to configure gRPC for multiple services in one ingress configuration. Locally it works perfect, but when I expose my services over ingress then I got the exception

# I use grpc.example.com:443 in my BloomRPC
{
  "error": "14 UNAVAILABLE: Trying to connect an http1.x server"
}

This is my ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: default
  name: ingress-grpc
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  tls:
  - hosts:
      - grpc.example.com
    secretName: grpc.example.com
  rules:
  - host: grpc.example.com
    http:
      paths:
      - path: /p3.protos.appservices.v1.AppService
        pathType: Prefix
        backend:
          service:
            name: app-service-svc
            port:
              name: grpc

kubectl describe ingress ingress-grpc

Name:             ingress-grpc
Namespace:        default
Address:          20.86.*.*
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  grpc.example.com terminates grpc.example.com
Rules:
  Host                       Path  Backends
  ----                       ----  --------
  grpc.example.com
                             /p3.protos.appservices.v1.AppService   app-service-svc:grpc (10.0.8.149:50051,10.0.8.59:50051)
Annotations:                 cert-manager.io/cluster-issuer: letsencrypt-prod
                             kubernetes.io/ingress.class: nginx
                             kubernetes.io/tls-acme: true
                             nginx.ingress.kubernetes.io/backend-protocol: GRPC
                             nginx.ingress.kubernetes.io/force-ssl-redirect: true
                             nginx.ingress.kubernetes.io/ssl-redirect: true
Events:
  Type    Reason             Age                   From                      Message
  ----    ------             ----                  ----                      -------
  Normal  CreateCertificate  84m                   cert-manager              Successfully created Certificate "grpc.example.com"
  Normal  Sync               54m (x55 over 5h47m)  nginx-ingress-controller  Scheduled for sync

And when I generate traffic with imported protos in BloomRPC you can see logs from NGINX something like this.

95.168.*.* - - [02/Nov/2021:14:13:32 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.084 [] [] - - - - a934d1ad915d451cc08e898c2160914e
95.168.*.* - - [02/Nov/2021:14:13:33 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.082 [] [] - - - - e1e80df41fe09295e9d2d21e9f581ac3
95.168.*.* - - [02/Nov/2021:14:13:33 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.082 [] [] - - - - 4eede56e8f9e43921f12d0385624f158
95.168.*.* - - [02/Nov/2021:14:13:33 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.082 [] [] - - - - 203d9672e2d6f04b1f91ae242d6bcd29
95.168.*.* - - [02/Nov/2021:14:13:34 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.092 [] [] - - - - a8c923716fda64c085e10ece2323f0e7
95.168.*.* - - [02/Nov/2021:14:13:34 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.342 [] [] - - - - 19ac57d82daabe11db0b664034a93719
95.168.*.* - - [02/Nov/2021:14:13:34 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.084 [] [] - - - - 0483af11f59e61996a00821cd85aa884
95.168.*.* - - [02/Nov/2021:14:13:35 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.352 [] [] - - - - 1e40a9bef77f0305d37e9f0a000dfd5c

My protos:p3.protos.appservices.v1.AppService.LocationStream

Regards

/kind bug

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 30 (14 by maintainers)

Most upvoted comments

I have a similar issue where i just see this in the logs: 35.153.65.211 - - [17/Aug/2022:16:26:49 +0000] "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.001 [] [] - - - - 1a706808c86a6c9c50e454459411b85

Did anyone find a solution?

+12
akashmantry on Aug 17, 2022

Yes @SalahBellagnaoui. It was a stupid thing. use-http2 was set to false in the configmap. Changing the value to true resolved it for me.

+1
akashmantry on Dec 23, 2022

So I tried setting up ingress-nginx with an ingress object like mentioned in this issue. I was using this helloworld example and by default the client in this case doesn’t have TLS setup by default and I got logs pretty similar to the OP’s description:

10.244.0.18 - - [26/Nov/2021:04:05:18 +0000] "_" "_" "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.000 [] [] - - - - ad71f4df1ca400bedc698c8447df64d4
10.244.0.18 - - [26/Nov/2021:04:05:24 +0000] "_" "_" "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.000 [] [] - - - - 3c2a82b060c98ef08d7870943f6ab763
10.244.0.18 - - [26/Nov/2021:04:05:25 +0000] "_" "_" "PRI * HTTP/2.0" 400 150 "-" "-" 0 0.000 [] [] - - - - 56bb0500786bdfda1d48d1058ad545ef

A little debugging in nginx revealed that TLS was the problem. So I added TLS to the grpc client like so:

 // Had to fork this function from grpc/credentials since I needed to set the “InsecureSkipVerify” flag for TLS
  func NewClientTLSFromFile(certFile, serverNameOverride string) (credentials.TransportCredentials, error) {
    b, err := ioutil.ReadFile(certFile)
    if err != nil {
      return nil, err 
    }
    cp := x509.NewCertPool()
    if !cp.AppendCertsFromPEM(b) {
      return nil, fmt.Errorf("credentials: failed to append certificates")
    }
    return credentials.NewTLS(&tls.Config{ServerName: serverNameOverride, RootCAs: cp, InsecureSkipVerify: true}), nil 
  }

  func main() {
    flag.Parse()
    // Set up a connection to the server.
    creds, err := NewClientTLSFromFile("/etc/ssl/certs/ca-certificates.crt", "grpc.example.com")
    if err != nil {
      log.Fatalf("Unable to create TLS creds")
    }
    conn, err := grpc.Dial(*addr, grpc.WithTransportCredentials(creds))
    if err != nil {
      log.Fatalf("did not connect: %v", err)
    }

And now the log looks like

10.244.0.25 - - [26/Nov/2021:05:11:55 +0000] "POST /helloworld.Greeter/SayHello HTTP/2.0" 503 190 "-" "grpc-go/1.41.0" 93 0.000 [test-ingress-service-port-50051-grpc] [] - - - - c7b7e33cf236c4c21f0b4400d69c0526
10.244.0.25 - - [26/Nov/2021:05:12:10 +0000] "POST /helloworld.Greeter/SayHello HTTP/2.0" 503 190 "-" "grpc-go/1.41.0" 93 0.000 [test-ingress-service-port-50051-grpc] [] - - - - 082366fbd2c4809da7148e5330c90ad3

which indicates a proper decode on ingress. The returned status is still 503 since the plumbing to the upstream server wasn’t correct in my case, but thats a different problem to solve.

+1
theunrealgeek on Nov 26, 2021

I think need to test non reflection api grpc on non kubernetes ingress, with just nginx as reverse proxy. If it works with non ingress and non kubernetes (vanilla nginx reverse proxy), then we can configure ingress controller potentially. @theunrealgeek for comments.

Thanks, ; Long

On Fri, 19 Nov, 2021, 1:37 PM Yury Kustov, @.***> wrote:

Could anybody make it clearer for me, should pure grpc (without reflection api etc) work over nginx-ingress-controller or only grpc-web is working? Thanks!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/7872#issuecomment-973847822, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWWG2UNAMHKTMWRTYHTUMYAUBANCNFSM5HGSH42A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

+1
longwuyuan on Nov 19, 2021

/remove-kind bug /kind support This example works https://kubernetes.github.io/ingress-nginx/examples/grpc/ so its has been documented.

Does this example work for you or does this example also fails for you ?

I follow this example, but I didn’t get any results.

+1
markofranjic on Nov 2, 2021
Read more comments on GitHub
← ingress-nginx: Global Config map not loaded even when explicitly defined in args and documentation on topic is severely lacking.
ingress-nginx: gRPC error handling →