kubernetes: PVCs do not get assigned to correct AZ as where the pod needs to be scheduled
/kind bug
What happened: I am trying to deploy a stateful set with PVCs to a set of 5 instances spread across 2 AZs. The stateful set has 5 pods, and I have set anti affinity so that there will be 1 pod per instance. The instances came up in AZs a-b-a-b-a. When I deploy the stateful set, their PVCs come up in AZs b-a-b-a-b. Thus the last pod in the set cannot be scheduled due to NoVolumeZoneConflict.
What you expected to happen: The PVCs should be assigned to the same AZ as the instance where the pod that requires it gets scheduled. Is there any way for me to make this happen?
How to reproduce it (as minimally and precisely as possible): Deploy a stateful set of 5 replicas with PVCs and anti pod affinity to a cluster with 5 nodes on 2 AZs:
spec:
replicas: 5
template:
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- foo
topologyKey: kubernetes.io/hostname
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 20Gi
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version
):
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1", GitCommit:"1dc5c66f5dd61da08412a74221ecc79208c2165b", GitTreeState:"clean", BuildDate:"2017-07-14T02:00:46Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T22:55:19Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration**: AWS
- OS (e.g. from /etc/os-release):
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
- Kernel (e.g.
uname -a
):
Linux 4.4.65-k8s x86_64 GNU/Linux
- Install tools: custom
- Others:
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 7
- Comments: 21 (10 by maintainers)
Please try https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode
Exactly, it’s mostly an issue when you are tight on resources. Unfortunately there are not really any good workarounds today.
I’m hoping to get PV scheduling to beta in 1.10. But it will still be at least another release before dynamic provisioning support will be added. Then, we can migrate existing zonal PVs to use it and solve this problem.
Ref kubernetes/features#490
PVC binding is currently done independently of pod scheduler, and is therefore not integrated with pod anti-affinity policies. I am working on making the PVC binding more integrated with pod scheduling, but it’s a big design change and will take a few releases to be fully functional.
/sig storage