descheduler: Panic with K8s v1.22: Close of closed channel

What version of descheduler are you using?

descheduler version: 0.23.0

Does this issue reproduce with the latest release?

yes

Which descheduler CLI options are you using?

          containers:
          - args:
            - --policy-config-file
            - /policy-dir/policy.yaml
            - --v
            - "3"
            command:
            - /bin/descheduler

Please provide a copy of your descheduler policy config file

apiVersion: v1
data:
  policy.yaml: |
    apiVersion: "descheduler/v1alpha1"
    kind: "DeschedulerPolicy"
    strategies:
      LowNodeUtilization:
        enabled: true
        params:
          nodeResourceUtilizationThresholds:
            targetThresholds:
              cpu: 50
              memory: 50
              pods: 50
            thresholds:
              cpu: 20
              memory: 20
              pods: 20
      RemoveDuplicates:
        enabled: true
      RemovePodsViolatingInterPodAntiAffinity:
        enabled: true
      RemovePodsViolatingNodeAffinity:
        enabled: true
        params:
          nodeAffinityType:
          - requiredDuringSchedulingIgnoredDuringExecution
      RemovePodsViolatingNodeTaints:
        enabled: true
      RemovePodsViolatingTopologySpreadConstraint:
        enabled: true
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: descheduler
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2021-09-14T08:04:24Z"
  labels:
    app.kubernetes.io/instance: descheduler
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: descheduler
    app.kubernetes.io/version: 0.23.0
    helm.sh/chart: descheduler-0.23.1
  name: descheduler
  namespace: kube-system
  resourceVersion: "281483144"
  uid: 21c26712-c359-4c6e-ad47-9f315c66fa1f

What k8s version are you using (kubectl version)?

kubectl version Output
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6", GitCommit:"f59f5c2fda36e4036b49ec027e556a15456108f0", GitTreeState:"clean", BuildDate:"2022-01-19T17:33:06Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6", GitCommit:"f59f5c2fda36e4036b49ec027e556a15456108f0", GitTreeState:"clean", BuildDate:"2022-01-19T17:26:47Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

What did you do?

Updated Kubernetes v1.19 to v1.22 (complying with Kubernetes Skew Policy). kube-descheduler sometimes runs without issues, but fails mostly:

descheduler-27416400--1-8nm4z              0/1     Error       0               5h5m    172.30.95.165    n0215   <none>           <none>
descheduler-27416400--1-dwlkn              0/1     Error       0               4h57m   172.30.95.176    n0215   <none>           <none>
descheduler-27416400--1-j5ns7              0/1     Error       0               5h4m    172.30.95.146    n0215   <none>           <none>
descheduler-27416400--1-jbw24              0/1     Error       0               5h5m    172.30.95.158    n0215   <none>           <none>
descheduler-27416400--1-nvqx2              0/1     Error       0               5h5m    172.30.95.154    n0215   <none>           <none>
descheduler-27416400--1-rx7l6              0/1     Error       0               5h5m    172.30.95.191    n0215   <none>           <none>
descheduler-27416400--1-v2v97              0/1     Error       0               5h3m    172.30.95.136    n0215   <none>           <none>
descheduler-27416680--1-5mw5k              0/1     Error       0               25m     172.30.95.184    n0215   <none>           <none>
descheduler-27416680--1-fmn7x              0/1     Completed   0               25m     172.30.95.160    n0215   <none>           <none>
descheduler-27416690--1-94v6h              0/1     Error       0               15m     172.30.95.173    n0215   <none>           <none>
descheduler-27416690--1-bbsnl              0/1     Error       0               15m     172.30.206.58    n0216   <none>           <none>
descheduler-27416690--1-bzk9h              0/1     Completed   0               15m     172.30.95.169    n0215   <none>           <none>
descheduler-27416700--1-42zcj              0/1     Error       0               5m41s   172.30.95.190    n0215   <none>           <none>
descheduler-27416700--1-5v6rx              0/1     Error       0               5m45s   172.30.206.47    n0216   <none>           <none>
descheduler-27416700--1-77547              0/1     Completed   0               5m31s   172.30.95.188    n0215   <none>           <none>

What did you expect to see?

No panic. 😎

What did you see instead?

I0216 08:50:02.917244       1 topologyspreadconstraint.go:170] "Skipping topology constraint because it is already balanced" constraint={MaxSkew:1 TopologyKey:topology.kubernetes.io/zone WhenUnsatisfiable:DoNotSchedule LabelSelector:&LabelSelector{MatchLabels:map[string]string{k8s-app: kube-dns,},MatchExpressions:[]LabelSelectorRequirement{},}}
I0216 08:50:02.917299       1 topologyspreadconstraint.go:170] "Skipping topology constraint because it is already balanced" constraint={MaxSkew:1 TopologyKey:topology.kubernetes.io/zone WhenUnsatisfiable:DoNotSchedule LabelSelector:&LabelSelector{MatchLabels:map[string]string{k8s-app: kube-dns,},MatchExpressions:[]LabelSelectorRequirement{},}}
I0216 08:50:02.917322       1 topologyspreadconstraint.go:170] "Skipping topology constraint because it is already balanced" constraint={MaxSkew:1 TopologyKey:topology.kubernetes.io/zone WhenUnsatisfiable:DoNotSchedule LabelSelector:&LabelSelector{MatchLabels:map[string]string{k8s-app: kube-dns,},MatchExpressions:[]LabelSelectorRequirement{},}}
I0216 08:50:03.044776       1 request.go:597] Waited for 127.315506ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/kubernetes-dashboard/pods
I0216 08:50:03.244906       1 request.go:597] Waited for 194.830034ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/log/pods
I0216 08:50:03.444566       1 request.go:597] Waited for 165.119862ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/metallb-system/pods
I0216 08:50:03.645101       1 request.go:597] Waited for 192.004131ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/pgadmin/pods
I0216 08:50:03.844799       1 request.go:597] Waited for 183.41437ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/postgres-operator/pods
I0216 08:50:04.044622       1 request.go:597] Waited for 194.921525ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/prometheus/pods
I0216 08:50:04.244880       1 request.go:597] Waited for 184.989867ms due to client-side throttling, not priority and fairness, request: GET:https://172.31.0.1:443/api/v1/namespaces/rook-ceph/pods
I0216 08:50:04.305330       1 descheduler.go:291] "Number of evicted pods" totalEvicted=0
I0216 08:50:04.305665       1 reflector.go:225] Stopping reflector *v1.Pod (0s) from k8s.io/client-go/informers/factory.go:134
panic: close of closed channel

goroutine 129 [running]:
sigs.k8s.io/descheduler/pkg/descheduler.Run.func1()
        /go/src/sigs.k8s.io/descheduler/pkg/descheduler/descheduler.go:76 +0x3c
created by sigs.k8s.io/descheduler/pkg/descheduler.Run
        /go/src/sigs.k8s.io/descheduler/pkg/descheduler/descheduler.go:74 +0x165

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 22 (22 by maintainers)

Commits related to this issue

Most upvoted comments

No panics anymore with the image mentioned in the comment above. Looks good! Thanks for your work!

Thanks, @damemi! I have pulled the image and let the descheduler run as CronJob for one or two days.

Wondering if we should publish a patch release for this… any thoughts?

Thanks for the update @stephan2012, we are thinking this will be fixed by https://github.com/kubernetes-sigs/descheduler/pull/731

thanks for clarify~ i am very clear now~~ sorry for my mistake for a long time