ingress-nginx: Upstream ExternalName services - proxy not working

I think I also have the same problem as https://github.com/kubernetes/ingress-nginx/issues/1332: I am seeing a 503 error in the nginx-controller log.

Here is the service which points to an AWS Elasticsearch available over https.

$kubectl get service/external-elasticsearch-service -n dev-andrew-0 -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2017-10-26T13:45:34Z
  name: external-elasticsearch-service
  namespace: dev-andrew-0
  resourceVersion: "11253256"
  selfLink: /api/v1/namespaces/dev-andrew-0/services/external-elasticsearch-service
  uid: f0a89a6c-ba53-11e7-9ff5-024b8c1b2a04
spec:
  externalName: search-es.eu-west-1.es.amazonaws.com
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}

Here is my ingress config dumped from kubectl. I trimmed out some stuff for readability

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  name: platform-frontend-dev-andrew-0
  namespace: dev-andrew-0
spec:
  rules:
  - host: dev-andrew.brickblock-dev.io
    http:
      paths:
      - backend:
          serviceName: external-elasticsearch-service
          servicePort: 443
        path: /

The nginx config from the ingress pod.

    server {
        server_name dev-andrew.foobar-dev.io;
        listen 80;
        listen [::]:80;
        set $proxy_upstream_name "-";

        vhost_traffic_status_filter_by_set_key $geoip_country_code country::$server_name;

        location / {
            set $proxy_upstream_name "
e";

            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;
            proxy_set_header X-Forwarded-For        $the_real_ip;
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   20s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

            proxy_pass http://dev-andrew-0-external-elasticsearch-service-443;
        }

    }

From the nginx pod the service does indeed seem to be available.

# curl -I http://dev-andrew-0-external-elasticsearch-service-443
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 340
Content-Type: application/json; charset=UTF-8
Connection: keep-alive

Here is an access log entry.

{
	"time": "2017-10-26T15:26:36+00:00",
	"status": "503",
	"remote_user": "-",
	"request_method": "GET",
	"server_protocol": "HTTP/1.1",
	"host": "dev-andrew.foobar-dev.io",
	"uri": "/",
	"request": "GET / HTTP/1.1",
	"request_id": "b0c5f68ae7d59f881d95211f57cbc05d",
	"args": "-",
	"request_length": "92",
	"request_time": "0.000",
	"bytes_sent": "390",
	"body_bytes_sent": "213",
	"http_referrer": "-",
	"http_user_agent": "curl/7.49.0",
	"upstream_addr": "127.0.0.1:8181",
	"upstream_response_length": "213",
	"upstream_response_time": "0.000",
	"upstream_status": "503",
	"proxy_upstream_name": "dev-andrew-0-external-elasticsearch-service-443",
	"proxy_protocol_addr": "",
	"proxy_add_x_forwarded_for": "100.96.22.0"
}

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 16 (10 by maintainers)

Most upvoted comments

Great! It’s working! NOTE - the 403 Forbidden response is coming from the external Elasticsearch and is a positive result.

ok, first please update the image to quay.io/aledbf/nginx-ingress-controller:0.269. This image contains the current master and PR https://github.com/kubernetes/ingress-nginx/pull/1605

This is the service:

  apiVersion: v1
  kind: Service
  metadata:
    name: external-elasticsearch-service
    namespace: dev-andrew-0
  spec:
    externalName: search-xxxxxxxxx.amazonaws.com
    type: ExternalName

and this the ingress:

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      ingress.kubernetes.io/secure-backends: "true"
      kubernetes.io/ingress.class: nginx
    name: platform-frontend-dev-andrew-0
    namespace: dev-andrew-0
  spec:
    rules:
    - host: xxxxxxxxxxxxx
      http:
        paths:
        - backend:
            serviceName: external-elasticsearch-service
            servicePort: 443
          path: /

the most important part here is the ingress.kubernetes.io/secure-backends: "true" annotation. (the externalname uses ssl)

this is the output

curl -v http://192.168.99.100:30507 -H 'Host: xxxxxxxxxxxxx'
* Rebuilt URL to: http://192.168.99.100:30507/
*   Trying 192.168.99.100...
* TCP_NODELAY set
* Connected to 192.168.99.100 (192.168.99.100) port 30507 (#0)
> GET / HTTP/1.1
> Host: xxxxxxxxxxxxx
> User-Agent: curl/7.55.1
> Accept: */*
> 
< HTTP/1.1 403 Forbidden
< Server: nginx/1.13.6
< Date: Fri, 27 Oct 2017 17:01:12 GMT
< Content-Type: application/json
< Content-Length: 99
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< x-amzn-RequestId: 6fb78426-bb38-11e7-813c-fba15c9040d5
< 
* Connection #0 to host 192.168.99.100 left intact
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: es-brick"}

@mooperd are you connected to the kubertenes slack channel?