kubernetes: When the number of jobs exceeds 500, cronjob cannot schedule, bug of pager.List

What happened: When the number of jobs exceeds 500, cronjob cannot schedule, bug of pager.List What you expected to happen: When the number of jobs exceeds 500, cronjobs are scheduled normally How to reproduce it (as minimally and precisely as possible): Create at least 501 jobs and one cronjob to see if cronjobs can be scheduled Anything else we need to know?: internalversion.List convert to batchv1.JobList error The following is the error log：

May 05 17:28:16 qa-cpu003.aibee.cn kube-controller-manager[40548]: E0505 17:28:16.100551   40548 cronjob_controller.go:117] expected type *batchv1.JobList, got type *internalversion.List

Environment:

Kubernetes version (use kubectl version): v1.14
Cloud provider or hardware configuration:
OS (e.g: cat /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Kernel (e.g. uname -a):

Linux qa-cpu001.aibee.cn 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Install tools: ansible and kube binary
Network plugin and version (if this is a network-related bug): calico
Others:

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 3
Comments: 23 (19 by maintainers)

Most upvoted comments

@jpbetz I have the same analyze about this bug. I push a pr to fix this. Would do like to review it?

	js := make([]batchv1.Job, 0)
	err = meta.EachListItem(jlTmp, func(object runtime.Object) error {
		jobTmp, ok := object.(*batchv1.Job)
		if !ok {
			return fmt.Errorf("Object must be a batchv1.Job.")
		}
		js = append(js, *jobTmp)
		return nil
	})

I try to use meta.EachListItem to change the list into []batchv1.Job cc @jingyih @smarterclayton

liucimin on May 6, 2019

As it stands, it looks like the pager.List API is error-prone, but I’m not certain how to fix it beyond improving the documentation (ideas?).

The cast to *batchv1.JobList in the cronjob controller is not safe. But it’s not obvious to developers since it works up to the page limit and intuitively seems like a reasonable thing to try.

pager.List returns a runtime.Object which can be either:

Type returned by the ListPageFunc provided to pager.New (in this case *batchv1.JobList), if the result fits in a single page
metainternalversion.List (see the pager.List implementation) if it exceeds a single page in size and must be accumulated.

meta.ListAccessor can be safely used to access the list returned by pager.List. As a starter, we should document that in pager.List and use it here. But we should also figure out how we can make pager.List safer for future use.

cc @jingyih @smarterclayton

jpbetz on May 6, 2019