kubernetes-ingress-controller: Sending requests to services using kong proxy internal service address results in invalid protocol error

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I’m running a kong ingress controller in a kubernetes cluster on openstack, where kic is installed in it’s own kong namespace and applications are deployed to a namespace services. I have configured ingresses for these apps in services namespace, and the external connections are fine. However when trying to communicate between applications through kong proxy service address and the exposed port 443, the requests fail and result in an invalid protocol error.

Expected Behavior

I’m expecting the applications to respond as intended.

Steps To Reproduce

Install kong ingress controller using helm3 with

helm upgrade king kong/kong --install --namespace kong --values values.yaml

Where values.yaml contains

image:
  repository: revomatico/docker-kong-oidc
  tag: 2.4.1-1

proxy:
  annotations:
    loadbalancer.openstack.org/floating-subnet: redacted
    loadbalancer.openstack.org/proxy-protocol: true
  externalTrafficPolicy: Local

replicaCount: 2

podDisruptionBudget:
  enabled: true
  maxUnavailable: "50%"

env:
  log_level: info
  nginx_proxy_large_client_header_buffers: "16 128k"
  proxy_buffer_size: "128k"
  anonymous_reports: off
  nginx_http_log_format: redacted
  nginx_http_lua_ssl_trusted_certificate: /etc/ssl/certs/ca-certificates.crt
  proxy_access_log: /dev/stdout laas
  proxy_listen: "0.0.0.0:8000 proxy_protocol, 0.0.0.0:8443 ssl proxy_protocol"
  real_ip_header: proxy_protocol
  trusted_ips: "0.0.0.0/0,::/0"
  x_session_compressor: zlib
  x_session_name: "oidc_session"
  nginx_proxy_proxy_busy_buffers_size: "256k"
  nginx_proxy_proxy_buffers: "16 128k"
  plugins: bundled,oidc
  ssl_cert: /etc/secrets/default-tls/tls.crt
  ssl_cert_key: /etc/secrets/default-tls/tls.key
  x_session_secret:
    valueFrom:
      secretKeyRef:
        name: kong-session-secret
        key: session-secret

ingressController:
  env:
    anonymous_reports: false
  installCRDs: false
  resources:
    requests:
      cpu: "200m"
      memory: "0.25Gi"
    limits:
      cpu: "500m"
      memory: "0.5Gi"

secretVolumes:
- default-tls

resources:
  requests:
    cpu: "200m"
    memory: "0.25Gi"
  limits:
    cpu: "500m"
    memory: "0.5Gi"

securityContext:
  runAsUser: 100
  fsGroup: 100

serviceMonitor:
  enabled: true

where the default-tls secret is a certificate provided by the kubernetes platform for respective dns available. The referenced kong-session-secret is created with

kubectl create secret generic kong-session-secret --namespace kong --from-literal=session-secret=$(openssl rand -base64 30)

For my application I would create ingresses as

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend-ingress
  namespace: services
  annotations:
    konghq.com/strip-path: "true"
    konghq.com/protocols: https
    konghq.com/https-redirect-status-code: "301"
spec:
  ingressClassName: kong
  tls:
  - hosts:
    - example.com
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend-app
            port:
              number: 8080

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: service-ingress
  namespace: services
  annotations:
    konghq.com/strip-path: "true"
    konghq.com/protocols: https
    konghq.com/https-redirect-status-code: "301"
    konghq.com/plugins: key-auth, basic-auth, services-acl
spec:
  ingressClassName: kong
  tls:
  - hosts:
    - example.com
  rules:
  - host: example.com
    http:
      paths:
      - path: /app-service
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 8080

Then try to call the app-service from the frontend application by using the created king-kong-proxy loadbalancer service. So the service address would be

king-kong-proxy.kong.svc.cluster.local:443/app-service

which results in invalid protocol. Same happened with the exposed http port :80. The plugins defined in the ingresses are custom kong-plugins implementing auth & acl’s.

Kong Ingress Controller version

1.3

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:38:26Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

Anything else?

No response