kubernetes: FailedScheduling didn't have free ports
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened: After deploy I have some pods in Pending state and they won’t change his state untill I restart kube-scheduler. kubectl describe pod <pod name>:
Warning FailedScheduling 20m (x745 over 24m) default-scheduler 0/72 nodes are available: 23 node(s) didn't match node selector, 43 node(s) didn't have free ports for the requested pod ports, 45 Insufficient cpu.
What you expected to happen: All pods should be in Running state after deployment
How to reproduce it (as minimally and precisely as possible):
kubectl apply -f test-deploy.yaml
kubectl apply -f test-deploy2.yaml
test-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-deploy
spec:
replicas: 25
selector:
matchLabels:
deployment: test-deploy
template:
metadata:
labels:
app: test-deploy
deployment: test-deploy
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: In
values: [ 'api-test' ]
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values: [ 'test-deploy']
topologyKey: kubernetes.io/hostname
tolerations:
- key: dedicated
operator: Equal
value: api-test
effect: NoExecute
containers:
- name: envoy-prometheus
image: example/envoy-exporter:latest
env:
- name: SERVER_PORT
value: "9092"
- name: URL
value: "http://127.0.0.1/stats"
ports:
- name: prom-stats
containerPort: 9092
imagePullSecrets:
- name: dockerhubkey
test-deploy2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-deploy2
spec:
replicas: 16
selector:
matchLabels:
deployment: test-deploy2
template:
metadata:
labels:
app: test-deploy
deployment: test-deploy2
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- 'app-pool2-1-europe-central1.testdc.com'
- 'app-pool2-2-europe-central1.testdc.com'
- 'app-pool2-3-europe-central1.testdc.com'
- 'app-pool2-4-europe-central1.testdc.com'
- 'app-pool2-5-europe-central1.testdc.com'
- 'app-pool2-6-europe-central1.testdc.com'
- 'app-pool2-7-europe-central1.testdc.com'
- 'app-pool2-8-europe-central1.testdc.com'
- 'app-pool2-9-europe-central1.testdc.com'
- 'app-pool2-10-europe-central1.testdc.com'
- 'app-pool2-11-europe-central1.testdc.com'
- 'app-pool2-12-europe-central1.testdc.com'
- 'app-pool2-13-europe-central1.testdc.com'
- 'app-pool2-14-europe-central1.testdc.com'
- 'app-pool2-15-europe-central1.testdc.com'
- 'app-pool2-16-europe-central1.testdc.com'
- 'app-pool2-17-europe-central1.testdc.com'
- 'app-pool2-18-europe-central1.testdc.com'
- 'app-pool2-19-europe-central1.testdc.com'
- 'app-pool2-20-europe-central1.testdc.com'
- 'app-pool2-21-europe-central1.testdc.com'
- 'app-pool2-22-europe-central1.testdc.com'
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values: [ 'test-deploy' ]
topologyKey: kubernetes.io/hostname
tolerations:
- key: dedicated
operator: Equal
value: api-test
effect: NoExecute
containers:
- name: envoy-prometheus
image: test/envoy-exporter:latest
env:
- name: SERVER_PORT
value: "9092"
- name: URL
value: "http://127.0.0.1/stats"
ports:
- name: prom-stats
containerPort: 9092
imagePullSecrets:
- name: dockerhubkey
Describe Pending pod
kubectl describe pod test-deploy2-675ccc77b7-2gmsm
Warning FailedScheduling 59s default-scheduler 0/72 nodes are available: 47 node(s) didn't have free ports for the requested pod ports, 54 node(s) didn't match node selector.
Restart kube-scheduler container
crictl stop $(crictl ps | grep kube-scheduler | awk '{print $1}')
Describe Pending pod
kubectl describe pod test-deploy2-675ccc77b7-2gmsm
Normal Scheduled 1m default-scheduler Successfully assigned default/test-deploy2-675ccc77b7-2gmsm to app-pool2-3-europe-central1.testdc.com
Anything else we need to know?:
I have two different deployments which use same Host ports but rise up on different nodes. It worked good when I used kubernetes v1.10 and it was broken when I updated to v.1.11
Environment:
- Kubernetes version (use
kubectl version): v1.11.0 - Cloud provider or hardware configuration: self-hosted AMD EPYC 7401P 24-Core 128 GB DDR4
- OS (e.g. from /etc/os-release): Ubuntu 18.04 LTS
- Kernel (e.g.
uname -a): 4.15.0-24-generic - Install tools: kubeadm
- Others: containerd v1.12, calico network v3.1
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 10
- Comments: 41 (19 by maintainers)
@jpiper thanks for the feedback. When you his the issue again, could you:
run the following command:
run
kubectl get po --all-namespaces -o wide | grep Pendingto list all Pending podsHi I faced with same issue on k3d after 2 years. 3 things need to be set:
If you have more nodes you need to specify @server:0,1,2 depends on your server count!
@Huang-Wei I just retried the scenario with 1.11.3 and
--feature-gates=PodPriority=trueon kube-scheduler and I could not reproduce the defect. So thanks again for your efforts, I think we can close the issue.I’m a colleague of @sfudeus. An observation I made is that I was initially unable to reproduce the issue (when I deleted the pods once) but when I “spammed” the delete command (
k delete pod -n metallb-system -l component=speaker) like 5 times directly after each other, the Pods began, as described in this issue, to not recover (stayed inPending). Though I’m not sure if the “spamming” of the command is actually a way to (always; I got it reproduced like 3-4 times in the cluster) reproduce the issue.So this was very odd. I have a 120 node cluster. I upgraded the control plane components to 1.11.1 and everything worked fine. When I upgraded all the node kubelets to 1.11.1 it stopped scheduling. When I downgraded the kubelets (but kept the control plane at 1.11.1) it still didn’t schedule.
When I then restarted the entire 1.11.1 control plane, it started working.
So I then upgraded all the kubelets to 1.11.1 and deleted a pod, and it was rescheduled fine. So, it looks like restarting all the control plane components fixed the issue, but I’m not sure why.
Here’s debug info whilst cluster is in a broken state (I have had to censor some things…)
I can see it’s been assinged to a node even though it’s stuck in pending
and the node 0626.corp.com only has the following ports seen by the master
@username1366 @jpiper also want to highlight that when you hit the issue, you don’t need to go to the physical node to check if the port is occupied (netstat/ss/etc.). Actually scheduler do not check real-time ports occupation, instead, it relies on the info from all running pods (those ones which uses
hostNetworkand openedcontainerPort) and make scheduling decisions.@username1366 from the log you provided, it seems the pod (failed to schedule) tried to claim ports 8080, 80, 443, 8001 and 9091 (from wantPorts info), and unfortunately some of them have been claimed to occupied - from existingPorts info:
So could you help to check which workloads have been using 8125, 19999, 8126. I know it’s irrelevant, but I’m thinking from these ports, it’s helpful to check if those workloads claims one of 8080, 80, 443, 8001 and 9091.