ingress-nginx: nginx ingress not working with grpc services

I am trying to get nginx ingress running as a Reverse proxy for my gRPC service.

my deployment.yml file looks like

          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: service-app
            labels:
              k8s-app: service-app
            namespace: service
          spec:
            replicas: 1
            selector:
              matchLabels:
                k8s-app: service-app
            template:
              metadata:
                labels:
                  k8s-app: service-app
              spec:
                containers:
                - name: service
                  image: "{{ ecr_repo }}/service:{{ git_id['stdout'] }}"
                  env:
                    - name: PORT
                      value: ":50051"
                  ports:
                  - containerPort: 50051
                    name: grpc

my svc.yml file looks like

         apiVersion: v1
         kind: Service
         metadata:
           name: service
           namespace: service
           labels:
             k8s-app: service-app
         spec:
           ports:
                 - port: 50051
                   targetPort: 50051
                   name: grpc
           selector:
             k8s-app: service-app

my ingress.yml file looks like

         apiVersion: networking.k8s.io/v1beta1
         kind: Ingress
         metadata:
             annotations:
                 kubernetes.io/ingress.class: "nginx"
                 nginx.ingress.kubernetes.io/ssl-redirect: "true"
                 nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
             name: service
             namespace: service
         spec:
             rules:
               - host: 
                 http:
                   paths:
                     - path:
                       backend:
                         serviceName: service
                         servicePort: grpc

When I do get ingress I get

NAME              HOSTS   ADDRESS       PORTS   AGE
service   *       10.0.213.16   80      9m44s

When I hit the alb arn with port 80, then I get 400 error similar to this

"PRI * HTTP/2.0" 400 157 "-" "-" 0 0.040 [] [] - - - - 9098e278e6c5ee7e1a2a4ecf02c6c2a1

Can any mistakes be pointed in the yml files of ingress deployment. It will be very helpful to get this thing running.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 21 (6 by maintainers)

Most upvoted comments

Me too. GRPC Web works though. A real pain

originswift-sys on May 27, 2021

May want to set a specific path value for the grpc service that aligns with the proto file. Currently you have an empty path value or path: /.

Examples:

- path: /build.stack.fortune.FortuneTeller
- path: /hello.HelloService

This allows the ingress to route the grpc requests specifically to the service.

In this case the ingress class annotations are not complete whereas here they are present.

May also need to add nginx.ingress.kubernetes.io/backend-protocol: "GRPCS" as indicated here.

a75c6 on Jul 23, 2020

I am getting the same issue

phuongTP1998 on May 24, 2021

@sbsends hey, Yes I am using AWS, and I haven’t made any progress on the issue, I cannot just make it work. Meanwhile, I just moved on to use Envoy with headless service in Kubernetes and it serves my purpose. For info on using Envoy you can refer here

Rahul7794 on Jul 22, 2020

@Rahul7794 Hey, we are having an identical issue on AWS. From the testing we have done, it seems like an AWS specific nginx deployment may be the problem (https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.1/deploy/static/provider/aws/deploy.yaml).

Are you using AWS?

Have you made any progress?

samsends on Jul 22, 2020

@Rahul7794 I cannot reproduce the issue

Create a cluster using kind: https://kind.sigs.k8s.io/docs/user/ingress/#create-cluster Install ingress-nginx: https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx

# change the IP address
export MY_IP=127.0.0.1

# create an SSL certificate
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout xip.key -out xip.crt -subj "/CN=$MY_IP.xip.io" \
  -addext "subjectAltName=DNS:$MY_IP.xip.io"

kubectl create secret tls xip-tls --cert=xip.crt --key=xip.key

echo "
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: grpcbin
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
spec:
  tls:
    - hosts:
      - $MY_IP.xip.io
      secretName: xip-tls
  rules:
  - host: $MY_IP.xip.io
    http:
      paths:
      - path: /
        backend:
          serviceName: grpcbin
          servicePort: 9000

---
kind: Pod
apiVersion: v1
metadata:
  name: grpcbin
  labels:
    app: grpcbin
spec:
  containers:
  - name: grpcbin
    image: moul/grpcbin

---
kind: Service
apiVersion: v1
metadata:
  name: grpcbin
spec:
  selector:
    app: grpcbin
  ports:
  - name: plain
    port: 9000
  - name: secured
    port: 9001

" | kubectl apply -f -

# https://github.com/fullstorydev/grpcurl
grpcurl -v -insecure $MY_IP.xip.io:443 hello.HelloService/SayHello

Resolved method descriptor:
{
  "name": "SayHello",
  "inputType": ".hello.HelloRequest",
  "outputType": ".hello.HelloResponse",
  "options": {
    
  }
}

Request metadata to send:
(empty)

Response headers received:
content-type: application/grpc
strict-transport-security: max-age=15724800; includeSubDomains
server: nginx/1.19.1
date: Tue, 14 Jul 2020 02:05:03 GMT


Response contents:
{
  "reply": "hello noname"
}

Response trailers received:
(empty)
Sent 0 requests and received 1 response

aledbf on Jul 14, 2020