istio: Getting 503 - Replicated Control Planes

Hi,

I have been following this doc: https://istio.io/latest/docs/setup/install/multicluster/gateways/ for setting up multi-cluster istio. I followed the same steps but I am getting 503 when using the curl command: kubectl exec -it sleep-6bdb595bcb-ssb58 -n foo -c sleep -- curl -I httpbin.bar.global:8000/headers

I used CoreDNS >= 1.4.0. I want to use Egress’s way of connecting two services. I tried both the hostname and IP address option of service entry.

For IP Address, I am doing nslookup for LB for istio-ingressgateway of httpbin cluster and putting the IP in place of ${CLUSTER2_GW_ADDR}

endpoints:
  - address: ${CLUSTER2_GW_ADDR}
    network: external
    ports:
      http1: 15443 # Do not change this port value
  - address: ${CLUSTER1_EGW_ADDR}
    ports:
      http1: 15443

For hostname, I am putting LB address in place of ${CLUSTER2_GW_ADDR}

endpoints:
  - address: ${CLUSTER2_GW_ADDR}
    network: external
    ports:
      http1: 15443
  - address: istio-egressgateway.istio-system.svc.cluster.local
    ports:
      http1: 15443

Both ways are not working and I am getting 503 errors. I am not sure how to debug this? Do, I need to create virtualservice/destination if httpbin?

Thanks

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 4
Comments: 39

Most upvoted comments

Hi. Same problem with “NR” logs in ingress as described by @PsycoSid

The same configuration works fine if we use 1.6.8 instead of 1.7.3.

So it really looks like a regression bug or maybe just outdated documentation. Anybody found the solution or a workaround or even a #bug number for this?

pupseba on Oct 8, 2020

I get the same with clusters in AWS is this consistent with it. When going from AWS to GCP it works. When going from GCP to AWS I get these 503 errors.

The connection debug logs show:-

2020-10-01T15:35:28.577148Z	debug	envoy connection	[C333] connecting to 15.236.241.203:15443
2020-10-01T15:35:28.577316Z	debug	envoy connection	[C333] connection in progress
2020-10-01T15:35:28.587518Z	debug	envoy connection	[C333] connected
2020-10-01T15:35:28.587622Z	debug	envoy connection	[C333] handshake expecting read
2020-10-01T15:35:28.598935Z	debug	envoy connection	[C333] handshake error: 5
2020-10-01T15:35:28.598964Z	debug	envoy connection	[C333]
2020-10-01T15:35:28.598969Z	debug	envoy connection	[C333] closing socket: 0
2020-10-01T15:35:28.643445Z	debug	envoy connection	[C334] connecting to 15.236.241.203:15443
2020-10-01T15:35:28.643568Z	debug	envoy connection	[C334] connection in progress
2020-10-01T15:35:28.653619Z	debug	envoy connection	[C334] connected
2020-10-01T15:35:28.653857Z	debug	envoy connection	[C334] handshake expecting read
2020-10-01T15:35:28.666264Z	debug	envoy connection	[C334] handshake error: 5
2020-10-01T15:35:28.666304Z	debug	envoy connection	[C334]
2020-10-01T15:35:28.666312Z	debug	envoy connection	[C334] closing socket: 0
2020-10-01T15:35:28.675526Z	debug	envoy connection	[C331] remote close
2020-10-01T15:35:28.675553Z	debug	envoy connection	[C331] closing socket: 0
2020-10-01T15:35:28.782013Z	debug	envoy connection	[C335] closing data_to_write=143 type=2
2020-10-01T15:35:28.782047Z	debug	envoy connection	[C335] setting delayed close timer with timeout 1000 ms
2020-10-01T15:35:28.782121Z	debug	envoy connection	[C335] write flush complete
2020-10-01T15:35:28.782453Z	debug	envoy connection	[C335] remote early close
2020-10-01T15:35:28.782483Z	debug	envoy connection	[C335] closing socket: 0
[2020-10-01T15:35:28.530Z] "GET /headers HTTP/1.1" 503 UF,URX "-" "-" 0 91 135 - "-" "curl/7.69.1" "db8a6217-7cfa-49c5-8e23-ab4c8143d132" "httpbin.bar.global:8000" "15.236.241.203:15443" outbound|8000||httpbin.bar.global - 240.0.0.2:8000 192.168.2.34:41610 - default

Where the Ingress GW logs seem to show a NR message for these:-

[2020-10-01T15:35:28.660Z] "- - -" 0 NR "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.2.145.193:15443 10.2.226.192:14400 - -
[2020-10-01T15:35:34.446Z] "- - -" 0 NR "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.2.145.193:15443 10.2.140.0:6300 - -
[2020-10-01T15:35:34.486Z] "- - -" 0 NR "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.2.145.193:15443 10.2.162.128:37898 - -
[2020-10-01T15:35:34.529Z] "- - -" 0 NR "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.2.145.193:15443 10.2.103.192:10141 - -

PsychoSid on Oct 1, 2020

Hi, I got the same issue. Below are images when I debugging Envoy and Istiod:

My sample config for 2 k8s clusters:

Cluster A:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    global:53 {
        errors
        cache 30
        forward . 10.107.133.68:53
    }
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-foo
spec:
  hosts:
  # must be of form name.namespace.global
  - httpbin.foo.global
  # Treat remote cluster services as part of the service mesh
  # as all clusters in the service mesh share the same root of trust.
  location: MESH_INTERNAL
  ports:
  - name: http1
    number: 8000
    protocol: http
  resolution: DNS
  addresses:
  - 240.0.0.2
  endpoints:
  # This is the routable address of the ingress gateway in cluster2 that
  # sits in front of sleep.foo service. Traffic from the sidecar will be
  # routed to this address.
  - address: 192.168.10.51
    ports:
      http1: 30538 # Do not change this port value

---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-bar
spec:
  hosts:
  # must be of form name.namespace.global
  - httpbin.bar.global
  # Treat remote cluster services as part of the service mesh
  # as all clusters in the service mesh share the same root of trust.
  location: MESH_INTERNAL
  ports:
  - name: http1
    number: 8000
    protocol: http
  resolution: DNS
  addresses:
  - 240.0.0.3
  endpoints:
  # This is the routable address of the ingress gateway in cluster2 that
  # sits in front of sleep.foo service. Traffic from the sidecar will be
  # routed to this address.
  - address: 192.168.10.51
    ports:
      http1: 30538 # Do not change this port value
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-legacy
spec:
  hosts:
  # must be of form name.namespace.global
  - httpbin.legacy.global
  # Treat remote cluster services as part of the service mesh
  # as all clusters in the service mesh share the same root of trust.
  location: MESH_INTERNAL
  ports:
  - name: http1
    number: 8000
    protocol: http
  resolution: DNS
  addresses:
  - 240.0.0.4
  endpoints:
  # This is the routable address of the ingress gateway in cluster2 that
  # sits in front of sleep.foo service. Traffic from the sidecar will be
  # routed to this address.
  - address: 192.168.10.51
    ports:
      http1: 30538 # Do not change this port value

Cluster B:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    global:53 {
        errors
        cache 30
        forward . 10.107.226.111:53
    }
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-foo
spec:
  hosts:
  # must be of form name.namespace.global
  - httpbin.foo.global
  # Treat remote cluster services as part of the service mesh
  # as all clusters in the service mesh share the same root of trust.
  location: MESH_INTERNAL
  ports:
  - name: http1
    number: 8000
    protocol: http
  resolution: DNS
  addresses:
  - 240.0.0.50
  endpoints:
  # This is the routable address of the ingress gateway in cluster2 that
  # sits in front of sleep.foo service. Traffic from the sidecar will be
  # routed to this address.
  - address: 192.168.10.50
    ports:
      http1: 32690 # Do not change this port value

---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-bar
spec:
  hosts:
  # must be of form name.namespace.global
  - httpbin.bar.global
  # Treat remote cluster services as part of the service mesh
  # as all clusters in the service mesh share the same root of trust.
  location: MESH_INTERNAL
  ports:
  - name: http1
    number: 8000
    protocol: http
  resolution: DNS
  addresses:
  - 240.0.0.51
  endpoints:
  # This is the routable address of the ingress gateway in cluster2 that
  # sits in front of sleep.foo service. Traffic from the sidecar will be
  # routed to this address.
  - address: 192.168.10.50
    ports:
      http1: 32690 # Do not change this port value
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-legacy
spec:
  hosts:
  # must be of form name.namespace.global
  - httpbin.legacy.global
  # Treat remote cluster services as part of the service mesh
  # as all clusters in the service mesh share the same root of trust.
  location: MESH_INTERNAL
  ports:
  - name: http1
    number: 8000
    protocol: http
  resolution: DNS
  addresses:
  - 240.0.0.52
  endpoints:
  # This is the routable address of the ingress gateway in cluster2 that
  # sits in front of sleep.foo service. Traffic from the sidecar will be
  # routed to this address.
  - address: 192.168.10.50
    ports:
      http1: 32690 # Do not change this port value

minhhoangvn on Aug 13, 2020

ok， Then I’ll let you know after I’ve tried. I think No problem And tell you how to configure it

clarechu on Jul 2, 2020

@kpiyush17 I do not using AWS Load Balancer URL, So this place didn’t help you

clarechu on Jul 4, 2020