istio: Internal Kubernetes API Calls Blocked by Istio

ok so I have another solution : try this master sidecar config: basically this will be injected to every istio managed POD . it will allow the pod to connect only to other PODs in the same namespace as well as connect to istio-system namespace and K8s API service

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: default
  namespace: istio-system  # or just place here your container NS
spec:  # also consider using a workloadSelector to fine tune your K8s API permissions to a specific pod
  egress:
    - hosts:
        - ./*
        - default/kubernetes.default.svc.cluster.local
        - default/etcd.default.svc.cluster.local
        - istio-system/*

credit not to me but to @WilliamNewshutz & @GregoryHanson

We have this problem - we made a ServiceEntry and VirtualService to account for the fact that the apiserver is now accessed over a public URL. I made the following ServiceEntry and VirtualService:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: azmk8s-ext
spec:
  hosts:
  - "<my-cluster>.hcp.centralus.azmk8s.io"
  location: MESH_EXTERNAL
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: tls-routing
spec:
  hosts:
  - <my-cluster>.hcp.centralus.azmk8s.io
  tls:
  - match:
    - port: 443
      sniHosts:
      - <my-cluster>.hcp.centralus.azmk8s.io
    route:
    - destination:
        host: <my-cluster>.hcp.centralus.azmk8s.io

This gets it to the point where I can access the api-server, but after 5 minutes, it stops working, and calls to the api-server hang. Also, calls to Go’s net.LookupIP(host) hang during this period, when providing the FQDN of the AKS apiserver.

I found that if I wait 10-15 minutes, the problem seems to resolve itself, but starts failing again after another 5 minutes. I also found that I can make a request to the apiserver when it’s working, and that seems to delay the point where it stops. I made a request, then another 1 minute later - and it started failing 5 minutes after the 2nd request, not the first.

I should mention that curl requests made directly to the API server when I kubectl exec into the pod DO succeed. That made me think that maybe it’s client-go that I was using incorrectly, but the fact that net.LookupIP(host) would also hang made me decide that probably isn’t the case…

We just switched from Contour to Istio (1.9.4) on our dev environments and are running into this issue alot

We’ve modified our IstioOperator settings with the excludeIPRanges mentioned in this issue, but we’re still seeing the issue.

It tends to happen regularly when our nightly builds get deployed to our dev clusters (~2AM PT). After enough restarts, the problem seems to go away, but we’ve yet to find the surefire workaround/remedy.

Any other things we should be looking at?

We managed to get around this issue with the following DestinationRul in our services’ namespace:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: k8s-default-destrule
  namespace: my-service-namespace
spec:
  host: "kubernetes.default.svc" #Disabling it for Kube API Server communication
  trafficPolicy:
    tls:
      mode: DISABLE

how about (lets assume kubernetes.default.svc.cluster.local = 172.21.0.1 since most cloud providers uses static IP range for K8s internals)

kind: IstioOperator
metadata:
  namespace: perators
  name: my-iop
spec:
  profile: empty
  values:
    global:
      proxy:
        excludeIPRanges: "172.21.0.0/32" # don't let istio egress rules block K8s API
        ... more code here

or if its only one POD which is affected: simply inject a POD annotation:

traffic.sidecar.istio.io/excludeOutboundIPRanges: "172.21.0.0/32" # don't let istio egress rules block K8s API

but the thing that works best for me is:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: k8s-api-ext
  namespace: istio-system 
spec:
  hosts:
    - kubernetes.default.svc.cluster.local
  addresses:
    - 172.21.0.1
  endpoints:
    - address: 172.21.0.1
  exportTo:
    - "*"
  location: MESH_EXTERNAL
  resolution: STATIC
  ports:
    - number: 443
      name: https-k8s
      protocol: HTTPS