descheduler: Panic with K8s v1.22: Close of closed channel
What version of descheduler are you using?
descheduler version: 0.23.0
Does this issue reproduce with the latest release?
yes
Which descheduler CLI options are you using?
containers:
- args:
- --policy-config-file
- /policy-dir/policy.yaml
- --v
- "3"
command:
- /bin/descheduler
Please provide a copy of your descheduler policy config file
apiVersion: v1
data:
policy.yaml: |
apiVersion: "descheduler/v1alpha1"
kind: "DeschedulerPolicy"
strategies:
LowNodeUtilization:
enabled: true
params:
nodeResourceUtilizationThresholds:
targetThresholds:
cpu: 50
memory: 50
pods: 50
thresholds:
cpu: 20
memory: 20
pods: 20
RemoveDuplicates:
enabled: true
RemovePodsViolatingInterPodAntiAffinity:
enabled: true
RemovePodsViolatingNodeAffinity:
enabled: true
params:
nodeAffinityType:
- requiredDuringSchedulingIgnoredDuringExecution
RemovePodsViolatingNodeTaints:
enabled: true
RemovePodsViolatingTopologySpreadConstraint:
enabled: true
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: descheduler
meta.helm.sh/release-namespace: kube-system
creationTimestamp: "2021-09-14T08:04:24Z"
labels:
app.kubernetes.io/instance: descheduler
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: descheduler
app.kubernetes.io/version: 0.23.0
helm.sh/chart: descheduler-0.23.1
name: descheduler
namespace: kube-system
resourceVersion: "281483144"
uid: 21c26712-c359-4c6e-ad47-9f315c66fa1f
What k8s version are you using (kubectl version
)?
kubectl version
Output
$ kubectl version Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6", GitCommit:"f59f5c2fda36e4036b49ec027e556a15456108f0", GitTreeState:"clean", BuildDate:"2022-01-19T17:33:06Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6", GitCommit:"f59f5c2fda36e4036b49ec027e556a15456108f0", GitTreeState:"clean", BuildDate:"2022-01-19T17:26:47Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
What did you do?
Updated Kubernetes v1.19 to v1.22 (complying with Kubernetes Skew Policy). kube-descheduler
sometimes runs without issues, but fails mostly:
descheduler-27416400--1-8nm4z 0/1 Error 0 5h5m 172.30.95.165 n0215 <none> <none>
descheduler-27416400--1-dwlkn 0/1 Error 0 4h57m 172.30.95.176 n0215 <none> <none>
descheduler-27416400--1-j5ns7 0/1 Error 0 5h4m 172.30.95.146 n0215 <none> <none>
descheduler-27416400--1-jbw24 0/1 Error 0 5h5m 172.30.95.158 n0215 <none> <none>
descheduler-27416400--1-nvqx2 0/1 Error 0 5h5m 172.30.95.154 n0215 <none> <none>
descheduler-27416400--1-rx7l6 0/1 Error 0 5h5m 172.30.95.191 n0215 <none> <none>
descheduler-27416400--1-v2v97 0/1 Error 0 5h3m 172.30.95.136 n0215 <none> <none>
descheduler-27416680--1-5mw5k 0/1 Error 0 25m 172.30.95.184 n0215 <none> <none>
descheduler-27416680--1-fmn7x 0/1 Completed 0 25m 172.30.95.160 n0215 <none> <none>
descheduler-27416690--1-94v6h 0/1 Error 0 15m 172.30.95.173 n0215 <none> <none>
descheduler-27416690--1-bbsnl 0/1 Error 0 15m 172.30.206.58 n0216 <none> <none>
descheduler-27416690--1-bzk9h 0/1 Completed 0 15m 172.30.95.169 n0215 <none> <none>
descheduler-27416700--1-42zcj 0/1 Error 0 5m41s 172.30.95.190 n0215 <none> <none>
descheduler-27416700--1-5v6rx 0/1 Error 0 5m45s 172.30.206.47 n0216 <none> <none>
descheduler-27416700--1-77547 0/1 Completed 0 5m31s 172.30.95.188 n0215 <none> <none>
What did you expect to see?
No panic. 😎
What did you see instead?
I0216 08:50:02.917244 1 topologyspreadconstraint.go:170] "Skipping topology constraint because it is already balanced" constraint={MaxSkew:1 TopologyKey:topology.kubernetes.io/zone WhenUnsatisfiable:DoNotSchedule LabelSelector:&LabelSelector{MatchLabels:map[string]string{k8s-app: kube-dns,},MatchExpressions:[]LabelSelectorRequirement{},}}
I0216 08:50:02.917299 1 topologyspreadconstraint.go:170] "Skipping topology constraint because it is already balanced" constraint={MaxSkew:1 TopologyKey:topology.kubernetes.io/zone WhenUnsatisfiable:DoNotSchedule LabelSelector:&LabelSelector{MatchLabels:map[string]string{k8s-app: kube-dns,},MatchExpressions:[]LabelSelectorRequirement{},}}
I0216 08:50:02.917322 1 topologyspreadconstraint.go:170] "Skipping topology constraint because it is already balanced" constraint={MaxSkew:1 TopologyKey:topology.kubernetes.io/zone WhenUnsatisfiable:DoNotSchedule LabelSelector:&LabelSelector{MatchLabels:map[string]string{k8s-app: kube-dns,},MatchExpressions:[]LabelSelectorRequirement{},}}
I0216 08:50:03.044776 1 request.go:597] Waited for 127.315506ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/kubernetes-dashboard/pods
I0216 08:50:03.244906 1 request.go:597] Waited for 194.830034ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/log/pods
I0216 08:50:03.444566 1 request.go:597] Waited for 165.119862ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/metallb-system/pods
I0216 08:50:03.645101 1 request.go:597] Waited for 192.004131ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/pgadmin/pods
I0216 08:50:03.844799 1 request.go:597] Waited for 183.41437ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/postgres-operator/pods
I0216 08:50:04.044622 1 request.go:597] Waited for 194.921525ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/prometheus/pods
I0216 08:50:04.244880 1 request.go:597] Waited for 184.989867ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/rook-ceph/pods
I0216 08:50:04.305330 1 descheduler.go:291] "Number of evicted pods" totalEvicted=0
I0216 08:50:04.305665 1 reflector.go:225] Stopping reflector *v1.Pod (0s) from k8s.io/client-go/informers/factory.go:134
panic: close of closed channel
goroutine 129 [running]:
sigs.k8s.io/descheduler/pkg/descheduler.Run.func1()
/go/src/sigs.k8s.io/descheduler/pkg/descheduler/descheduler.go:76 +0x3c
created by sigs.k8s.io/descheduler/pkg/descheduler.Run
/go/src/sigs.k8s.io/descheduler/pkg/descheduler/descheduler.go:74 +0x165
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 22 (22 by maintainers)
Commits related to this issue
- Change deschduler to deployment Attempt to resolve issue https://github.com/kubernetes-sigs/descheduler/issues/728 — committed to dexter136/homelab by dexter136 2 years ago
No panics anymore with the image mentioned in the comment above. Looks good! Thanks for your work!
Thanks, @damemi! I have pulled the image and let the descheduler run as CronJob for one or two days.
Wondering if we should publish a patch release for this… any thoughts?
Thanks for the update @stephan2012, we are thinking this will be fixed by https://github.com/kubernetes-sigs/descheduler/pull/731
thanks for clarify~ i am very clear now~~ sorry for my mistake for a long time