kubernetes: publishNotReadyAddresses not working on service

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened: Set up a headless service with publishNotReadyAddresses set to true. Set up a stateful set that uses that service as its governing service, and specified a readiness check. The pods come up, but never move to ready, and the pods also never show up as endpoints in the service.

What you expected to happen: The pods should be able to talk to each other through the service, even though they are not Ready.

How to reproduce it (as minimally and precisely as possible): This is a bit tricky, but it should work (fail) if:

  1. create a service with publishNotReadyAddresses: true
  2. create a stateful set with 3 pods, and podManagementPolicy: "Parallel",
  3. specify a readiness check that always fails
  4. observe that the pods come up, but cannot talk to each other and are not listed as endpoints in the service.

Anything else we need to know?: I’ve also set "service.beta.kubernetes.io/tolerate-unready-endpoints": "yes", just in case, but to no avail.

Also, if I remove the readiness check, everything works fine, so I’m pretty sure it’s not some other miss-configuration causing this.

Here’s the output of kubectl describe on the service:

Namespace:         hc-vault
Labels:            <none>
Annotations:       kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.beta.kubernetes.io/tolerate-unready-endpoints":"yes"},"name":"consul-internal",...
                   service.beta.kubernetes.io/tolerate-unready-endpoints=yes
Selector:          app=consul
Type:              ClusterIP
IP:                None
Port:              http  8500/TCP
TargetPort:        8500/TCP
Endpoints:         
Port:              rpc  8400/TCP
TargetPort:        8400/TCP
Endpoints:         
Port:              serflan-tcp  8301/TCP
TargetPort:        8301/TCP
Endpoints:         
Port:              serflan-udp  8301/UDP
TargetPort:        8301/UDP
Endpoints:         
Port:              serfwan-tcp  8302/TCP
TargetPort:        8302/TCP
Endpoints:         
Port:              sefwan-udp  8302/UDP
TargetPort:        8302/UDP
Endpoints:         
Port:              server  8300/TCP
TargetPort:        8300/TCP
Endpoints:         
Port:              consuldns  8600/TCP
TargetPort:        8600/TCP
Endpoints:         
Port:              statsd-scrape  9102/TCP
TargetPort:        9102/TCP
Endpoints:         
Session Affinity:  None
Events:            <none>

and kubectl get pods

NAME                          READY     STATUS    RESTARTS   AGE
consul-0                      1/2       Running   0          17m
consul-1                      1/2       Running   0          17m
consul-2                      1/2       Running   0          17m

Environment:

  • Kubernetes version (use kubectl version): 1.8.1
  • Cloud provider or hardware configuration: azure
  • OS (e.g. from /etc/os-release): CoreOS
  • Kernel (e.g. uname -a): Linux 4.13.9-coreos #1 SMP
  • Install tools:
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 7
  • Comments: 16 (5 by maintainers)

Commits related to this issue

Most upvoted comments

Hi @hobti01, the publishNotReadyAddresses implementation was done recently. I don’t see it in Kubernetes 1.10, but I do see it in 1.11, which may explain the behavior you saw.

This issue has been fixed in master. See https://github.com/kubernetes/kubernetes/pull/63742

@maxboisvert sorry, if you’re looking for a repro case: remove the annotation from the service metadata, and add publishNotReadyAddresses: true to the service spec.