kubernetes: publishNotReadyAddresses not working on service
Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
/kind bug
/kind feature
What happened:
Set up a headless service with publishNotReadyAddresses
set to true. Set up a stateful set that uses that service as its governing service, and specified a readiness check. The pods come up, but never move to ready, and the pods also never show up as endpoints in the service.
What you expected to happen: The pods should be able to talk to each other through the service, even though they are not Ready.
How to reproduce it (as minimally and precisely as possible): This is a bit tricky, but it should work (fail) if:
- create a service with
publishNotReadyAddresses: true
- create a stateful set with 3 pods, and
podManagementPolicy: "Parallel",
- specify a readiness check that always fails
- observe that the pods come up, but cannot talk to each other and are not listed as endpoints in the service.
Anything else we need to know?:
I’ve also set "service.beta.kubernetes.io/tolerate-unready-endpoints": "yes",
just in case, but to no avail.
Also, if I remove the readiness check, everything works fine, so I’m pretty sure it’s not some other miss-configuration causing this.
Here’s the output of kubectl describe
on the service:
Namespace: hc-vault
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.beta.kubernetes.io/tolerate-unready-endpoints":"yes"},"name":"consul-internal",...
service.beta.kubernetes.io/tolerate-unready-endpoints=yes
Selector: app=consul
Type: ClusterIP
IP: None
Port: http 8500/TCP
TargetPort: 8500/TCP
Endpoints:
Port: rpc 8400/TCP
TargetPort: 8400/TCP
Endpoints:
Port: serflan-tcp 8301/TCP
TargetPort: 8301/TCP
Endpoints:
Port: serflan-udp 8301/UDP
TargetPort: 8301/UDP
Endpoints:
Port: serfwan-tcp 8302/TCP
TargetPort: 8302/TCP
Endpoints:
Port: sefwan-udp 8302/UDP
TargetPort: 8302/UDP
Endpoints:
Port: server 8300/TCP
TargetPort: 8300/TCP
Endpoints:
Port: consuldns 8600/TCP
TargetPort: 8600/TCP
Endpoints:
Port: statsd-scrape 9102/TCP
TargetPort: 9102/TCP
Endpoints:
Session Affinity: None
Events: <none>
and kubectl get pods
NAME READY STATUS RESTARTS AGE
consul-0 1/2 Running 0 17m
consul-1 1/2 Running 0 17m
consul-2 1/2 Running 0 17m
Environment:
- Kubernetes version (use
kubectl version
): 1.8.1 - Cloud provider or hardware configuration: azure
- OS (e.g. from /etc/os-release): CoreOS
- Kernel (e.g.
uname -a
): Linux 4.13.9-coreos #1 SMP - Install tools:
- Others:
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 7
- Comments: 16 (5 by maintainers)
Commits related to this issue
- publishNotReadyAddresses: true does not actually work. https://github.com/kubernetes/kubernetes/issues/58662 — committed to maratoid/chart-vault by maratoid 6 years ago
- Add readiness probe. Add populating secrets on setup (#90) * remove hardcoded node selector * add readiness probe * add readiness check. internally use a headless service that skips readiness c... — committed to samsung-cnct/chart-vault by maratoid 6 years ago
- cockroachdb: Switch from unready-endpoints annotation to proper field This works as far back as Kubernetes 1.8, which means we're still supporting the 3 most recently release versions of Kubernetes, ... — committed to a-robinson/charts by a-robinson 6 years ago
- kubernetes: Continue providing tolerate-unready-endpoints annotation This partially walks back #23475 because I found while updating our Chart that the proper field isn't actually respected by Kubern... — committed to a-robinson/cockroach by a-robinson 6 years ago
- Merge #24643 #24644 24643: kubernetes: Continue providing tolerate-unready-endpoints annotation r=a-robinson a=a-robinson This partially walks back #23475 because I found while updating our Chart t... — committed to cockroachdb/cockroach by deleted user 6 years ago
- cockroachdb: Various improvements (#4876) * cockroachdb: Add readiness/liveness probes * cockroachdb: Switch from unready-endpoints annotation to proper field This works as far back as Kubernetes 1... — committed to helm/charts by a-robinson 6 years ago
- cockroachdb: Various improvements (#4876) * cockroachdb: Add readiness/liveness probes * cockroachdb: Switch from unready-endpoints annotation to proper field This works as far back as Kubernetes 1... — committed to Bestmile/charts by a-robinson 6 years ago
- cockroachdb: Various improvements (#4876) * cockroachdb: Add readiness/liveness probes * cockroachdb: Switch from unready-endpoints annotation to proper field This works as far back as Kubernetes 1... — committed to dysnix/helm-charts by a-robinson 6 years ago
- publishNotReadyAddresses not working according to https://github.com/kubernetes/kubernetes/issues/58662 the `publishNotReadyAddresses: true` is not working. Instead they suggest this annotation as wo... — committed to MichaelSp/helm-charts by MichaelSp 5 years ago
- publishNotReadyAddresses not working according to https://github.com/kubernetes/kubernetes/issues/58662 the `publishNotReadyAddresses: true` is not working. Instead they suggest this annotation as wo... — committed to mtp-devops/3d-party-helm by MichaelSp 5 years ago
Hi @hobti01, the
publishNotReadyAddresses
implementation was done recently. I don’t see it in Kubernetes 1.10, but I do see it in 1.11, which may explain the behavior you saw.This issue has been fixed in master. See https://github.com/kubernetes/kubernetes/pull/63742
@maxboisvert sorry, if you’re looking for a repro case: remove the annotation from the service metadata, and add
publishNotReadyAddresses: true
to the service spec.