istio: Headless service broken in 1.6.0 (no healthy upstream)

Upgrading from 1.5.4 to 1.6.0 resulted in “no healthy upstream” errors from Envoy. It worked perfectly when rolling back to 1.5.4

Finally resolved the issue by removing clusterIP: None from the affected services. Seems 1.5.4 was Ok with headless services: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services

But 1.6.0 is not. Not sure if really a bug, or working as expected, but I thought I’d put the issue in for visibility.

[X] Networking

Expected behavior Envoy to work with headless services

Steps to reproduce the bug Create a “headless” service with clusterIP: None. This will still work with an ingress controller, and works with Istio 1.5.4 but not since upgrading to 1.6.0

How was Istio installed? istioctl manifest apply --set values.kiali.enabled=true --set values.gateways.istio-ingressgateway.type=NodePort

Environment where bug was observed (cloud vendor, OS, etc) “Bare metal” kubeadm deployment 1.18.3

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 7
Comments: 23 (8 by maintainers)

Most upvoted comments

@hzxuzhonghu @incfly @howardjohn can I understand that the correct behavious/configuration are as below when mtls.auto=true, mtls.enabled=false in global, please help to confirm if it is what we want finally.

not need to update env PILOT_ENABLE_EDS_FOR_HEADLESS_SERVICES (false as default) and PILOT_ENABLE_HEADLESS_SERVICE_POD_LISTENERS (true as default)
not need to update headless service port name as <protocol>-xxx (for Iuse legacy service which does not follow istio port name requirement.)
when headless service WITH sidecar injected, 3.1 if wants to use plaintext between client and headless service, need to setup PeerAuthentication with “mtls.mode: DISABLE” or DR with tls.mode: DISABLE 3.2 if want to use mtls, need to setup DR with ISTIO_MUTUAL. one more question here: which one is default behavious?
when headless service WITHOUT sidecar injected. 4.1 if want to use plaintext between client and headless service, need to setup PeerAuthentication with “mtls.mode: DISABLE” or DR with tls.mode: DISABLE 4.2 for headless service is not in-mesh, so not in-mesh mtls here. one more question here: which one is default behavious?

My team tests case 4.1 with DR(DISABLE) in 1.7.2 with headless service is kafka, and case 3.2 with DR (ISTIO_MUTUAL) and headless service is database in 1.6.7, and both work.

The PeerAuthentication and DR configuration from hzxuzhonghu I list as below for reference only.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: xxxx
spec:
  host: headless-service
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL/DISABLE

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
spec:
  selector:
    matchLabels:
      app: headlessxxx
      version: v1
  mtls:
    mode: DISABLE

hobbytp on Oct 14, 2020

@4qv907rtet5r @jshum2479 @so-jelly

I resolved my problem by set PILOT_ENABLE_EDS_FOR_HEADLESS_SERVICES as true and PILOT_ENABLE_HEADLESS_SERVICE_POD_LISTENERS as false in istiod which in version 1.6.5.

I found them in https://istio.io/latest/docs/reference/commands/pilot-agent/#envvars

After this, i found those headless services in istioctl pc endpoints and i can request them successfully in the mesh.

suninuni on Jul 15, 2020

Since 1.7, headless+auto mtls should work in all cases.

https://github.com/istio/istio.io/issues/8393 tracks better docs 1.6 is EOL https://istio.io/latest/news/support/announcing-1.6-eol/

So I think there is no action left here

howardjohn on Nov 24, 2020