istio: Mutual TLS not working (ISTIO_MUTUAL is ignored)

Describe the bug Following the tutorial: https://istio.io/docs/tasks/security/authn-policy/ When enabling istio mutual auth, the services did get 503, when creating the destination rule with ISTIO_MUTUAL - still, 503 errors

Attaching also the debug logs of the sidecar of sleep: sleep_sidecar.txt

Strangely, when added the rule

kubectl apply -f - <<EOF
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "foo"
spec:
  host: "*.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
EOF

foo managed to go to bar, but not to himself:

$ for from in "foo" "bar"; do for to in "foo" "bar"; do kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl "http://httpbin.${to}:8000/ip" -s -o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}\n"; done; done

sleep.foo to httpbin.foo: 503
sleep.foo to httpbin.bar: 200
sleep.bar to httpbin.foo: 503
sleep.bar to httpbin.bar: 503

Expected behavior like in the tutorial, should get 200 from services

Steps to reproduce the bug Follow the tutorial on AWS EKS

Version Kubernetes Client Version: version.Info{Major:“1”, Minor:“10”, GitVersion:“v1.10.3”, GitCommit:“2bba0127d85d5a46ab4b778548be28623b32d0b0”, GitTreeState:“clean”, BuildDate:“2018-07-26T20:40:11Z”, GoVersion:“go1.9.3”, Compiler:“gc”, Platform:“windows/amd64”} Server Version: version.Info{Major:“1”, Minor:“10+”, GitVersion:“v1.10.13-eks-g484b8”, GitCommit:“484b857e3134d55ac6373fea2f51798fefa0533f”, GitTreeState:“clean”, BuildDate:“2019-03-08T05:32:36Z”, GoVersion:“go1.10.8”, Compiler:“gc”, Platform:“linux/amd64”}

Istio client version: version.BuildInfo{Version:“1.1.0”, GitRevision:“82797c0c0649a3f73029b33957ae105260458c6e”, User:“root”, Host:“996cd064-49c1-11e9-813c-0a580a2c0506”, GolangVersion:“go1.10.4”, DockerHub:“docker.io/istio”, BuildStatus:“Clean”, GitTag:“1.1.0-rc.6”} citadel version: version.BuildInfo{Version:“1.1.0”, GitRevision:“82797c0c0649a3f73029b33957ae105260458c6e-dirty”, User:“root”, Host:“1f4e9557-49cd-11e9-86e9-0a580a2c0304”, GolangVersion:“go1.10.4”, DockerHub:“docker.io/istio”, BuildStatus:“Modified”, GitTag:“1.1.0-rc.6”} egressgateway version: version.BuildInfo{Version:“1.1.0”, GitRevision:“82797c0c0649a3f73029b33957ae105260458c6e”, User:“root”, Host:“1f4e9557-49cd-11e9-86e9-0a580a2c0304”, GolangVersion:“go1.10.4”, DockerHub:“docker.io/istio”, BuildStatus:“Clean”, GitTag:“1.1.0-rc.6”} galley version: version.BuildInfo{Version:“1.1.0”, GitRevision:“82797c0c0649a3f73029b33957ae105260458c6e-dirty”, User:“root”, Host:“1f4e9557-49cd-11e9-86e9-0a580a2c0304”, GolangVersion:“go1.10.4”, DockerHub:“docker.io/istio”, BuildStatus:“Modified”, GitTag:“1.1.0-rc.6”} ingressgateway version: version.BuildInfo{Version:“1.1.0”, GitRevision:“82797c0c0649a3f73029b33957ae105260458c6e”, User:“root”, Host:“1f4e9557-49cd-11e9-86e9-0a580a2c0304”, GolangVersion:“go1.10.4”, DockerHub:“docker.io/istio”, BuildStatus:“Clean”, GitTag:“1.1.0-rc.6”} pilot version: version.BuildInfo{Version:“1.1.0”, GitRevision:“82797c0c0649a3f73029b33957ae105260458c6e-dirty”, User:“root”, Host:“1f4e9557-49cd-11e9-86e9-0a580a2c0304”, GolangVersion:“go1.10.4”, DockerHub:“docker.io/istio”, BuildStatus:“Modified”, GitTag:“1.1.0-rc.6”} policy version: version.BuildInfo{Version:“1.1.0”, GitRevision:“82797c0c0649a3f73029b33957ae105260458c6e-dirty”, User:“root”, Host:“1f4e9557-49cd-11e9-86e9-0a580a2c0304”, GolangVersion:“go1.10.4”, DockerHub:“docker.io/istio”, BuildStatus:“Modified”, GitTag:“1.1.0-rc.6”} sidecar-injector version: version.BuildInfo{Version:“1.1.0”, GitRevision:“82797c0c0649a3f73029b33957ae105260458c6e-dirty”, User:“root”, Host:“1f4e9557-49cd-11e9-86e9-0a580a2c0304”, GolangVersion:“go1.10.4”, DockerHub:“docker.io/istio”, BuildStatus:“Modified”, GitTag:“1.1.0-rc.6”} telemetry version: version.BuildInfo{Version:“1.1.0”, GitRevision:“82797c0c0649a3f73029b33957ae105260458c6e-dirty”, User:“root”, Host:“1f4e9557-49cd-11e9-86e9-0a580a2c0304”, GolangVersion:“go1.10.4”, DockerHub:“docker.io/istio”, BuildStatus:“Modified”, GitTag:“1.1.0-rc.6”}

Installation EKS environment as mentioned, 3 worker nodes:

kubectl create -f install/kubernetes/helm/helm-service-account.yaml
helm template install/kubernetes/helm/istio --name istio --namespace istio-system --set grafana.enabled=true --set servicegraph.enabled=true > istio.yaml
kubectl create namespace istio-system
kubectl apply -f istio.yaml

then applied install/kubernetes/istio-demo.yaml: kubectl apply -f install/kubernetes/istio-demo.yaml

Environment EKS environment (AWS)

Cluster state dump.zip

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 30 (9 by maintainers)

Most upvoted comments

It’s no activity by an Istio team member. I’ll update the bot’s comment text accordingly to make it clearer.

geeknoid on Nov 6, 2019

Thanks for the report. This is because we change the order to select the destination rule to be applied. Long story short, Istio will pick the DR in the same namespace of the client app first, then the one in the server namespace, and finally the one in the global namespace (by default, istio-system). As a result, the DR under default namespace is not visible to services in foo or bar. I’ll update the documentation. In the meantime, can you change the DR rules to use istio-system namespace instead of default . Alternatively, you can submit the DR to both foo and bar namespace.

diemtvu on Mar 26, 2019