ingress-nginx: Upstream ExternalName services - proxy not working
I think I also have the same problem as https://github.com/kubernetes/ingress-nginx/issues/1332: I am seeing a 503 error in the nginx-controller log.
Here is the service which points to an AWS Elasticsearch available over https.
$kubectl get service/external-elasticsearch-service -n dev-andrew-0 -o yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2017-10-26T13:45:34Z
name: external-elasticsearch-service
namespace: dev-andrew-0
resourceVersion: "11253256"
selfLink: /api/v1/namespaces/dev-andrew-0/services/external-elasticsearch-service
uid: f0a89a6c-ba53-11e7-9ff5-024b8c1b2a04
spec:
externalName: search-es.eu-west-1.es.amazonaws.com
sessionAffinity: None
type: ExternalName
status:
loadBalancer: {}
Here is my ingress config dumped from kubectl. I trimmed out some stuff for readability
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
name: platform-frontend-dev-andrew-0
namespace: dev-andrew-0
spec:
rules:
- host: dev-andrew.brickblock-dev.io
http:
paths:
- backend:
serviceName: external-elasticsearch-service
servicePort: 443
path: /
The nginx config from the ingress pod.
server {
server_name dev-andrew.foobar-dev.io;
listen 80;
listen [::]:80;
set $proxy_upstream_name "-";
vhost_traffic_status_filter_by_set_key $geoip_country_code country::$server_name;
location / {
set $proxy_upstream_name "
e";
port_in_redirect off;
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 20s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_redirect off;
proxy_buffering off;
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
proxy_pass http://dev-andrew-0-external-elasticsearch-service-443;
}
}
From the nginx pod the service does indeed seem to be available.
# curl -I http://dev-andrew-0-external-elasticsearch-service-443
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Length: 340
Content-Type: application/json; charset=UTF-8
Connection: keep-alive
Here is an access log entry.
{
"time": "2017-10-26T15:26:36+00:00",
"status": "503",
"remote_user": "-",
"request_method": "GET",
"server_protocol": "HTTP/1.1",
"host": "dev-andrew.foobar-dev.io",
"uri": "/",
"request": "GET / HTTP/1.1",
"request_id": "b0c5f68ae7d59f881d95211f57cbc05d",
"args": "-",
"request_length": "92",
"request_time": "0.000",
"bytes_sent": "390",
"body_bytes_sent": "213",
"http_referrer": "-",
"http_user_agent": "curl/7.49.0",
"upstream_addr": "127.0.0.1:8181",
"upstream_response_length": "213",
"upstream_response_time": "0.000",
"upstream_status": "503",
"proxy_upstream_name": "dev-andrew-0-external-elasticsearch-service-443",
"proxy_protocol_addr": "",
"proxy_add_x_forwarded_for": "100.96.22.0"
}
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 16 (10 by maintainers)
Great! It’s working! NOTE - the 403 Forbidden response is coming from the external Elasticsearch and is a positive result.
ok, first please update the image to
quay.io/aledbf/nginx-ingress-controller:0.269. This image contains the current master and PR https://github.com/kubernetes/ingress-nginx/pull/1605This is the service:
and this the ingress:
the most important part here is the
ingress.kubernetes.io/secure-backends: "true"annotation. (the externalname uses ssl)this is the output
@mooperd are you connected to the kubertenes slack channel?