kubernetes: Daemonsets pods are being scheduled with duplicate IPs
What happened: Pods are being scheduled with duplicate IPs
What you expected to happen: Unique IP per pods
How to reproduce it (as minimally and precisely as possible): No clue, it seems to happen randomly. I only noticed it when my errors rate went up, and one of my Nginx Ingress (daemonset) wasn’t reachable because of the IP conflict.
Most of the IP conflict remaining are bad but it seems like the cluster is still humming
kube-system ip-masq-agent-wsqn5 1/1 Running 0 15h 172.20.1.10 gke-plr-web-prd-05-web-00-531bb1cd-2djt <none>
kube-system kube-proxy-gke-plr-web-prd-05-web-00-531bb1cd-2djt 1/1 Running 0 15h 172.20.1.10 gke-plr-web-prd-05-web-00-531bb1cd-2djt <none>
monitoring node-exporter-mz76k 1/1 Running 0 15h 172.20.1.10 gke-plr-web-prd-05-web-00-531bb1cd-2djt <none>
Anything else we need to know?: I created a GKE cluster with a custom pod address range and VPC-Native on. It looks like a race condition thing because it seems to only happen to DaemonSets and system pods, and the conflict is all on the same node, and the pod all have the same age
Environment:
- Kubernetes version (use
kubectl version
):
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-08T16:31:10Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.2-gke.18", GitCommit:"5796233393d7bc034428de15191ad3d2eaff95fb", GitTreeState:"clean", BuildDate:"2018-11-08T20:49:08Z", GoVersion:"go1.10.3b4", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration:
GKE
Master version => 1.11.2-gke.18
Endpoint => <redacted>
Client certificate => Enabled
Binary authorization => Disabled
Kubernetes alpha features => Disabled
Current total size => 33
Master zone => us-west1-a
Node zones => us-west1-a
Network => <redacted>
Subnet => <redacted>
VPC-native (alias IP) => Enabled
Pod address range => 172.30.0.0/17
Default maximum pods per node => 110
Service address range => 172.30.128.0/17
Stackdriver Logging => Enabled
Stackdriver Monitoring => Enabled
Private cluster => Disabled
Master authorized networks => Disabled
Network policy => Disabled
Legacy authorization => Disabled
Maintenance window => 3:00 PM
Cloud TPU => Disabled
Node auto-provisioning => Disabled
- OS (e.g. from /etc/os-release):
- Kernel (e.g.
uname -a
): - Install tools:
- Others:
/kind bug
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 20 (13 by maintainers)
Just a quote from k8s documentation:
So this could be source of the confusion - as we all learned here: it does not have be always true 😀