kubernetes: Updating a cronjob causes jobs to be scheduled retroactively
/kind bug
What happened: When updating a cronjob, the schedule takes effect retroactively, meaning “missed” jobs from before the update are started What you expected to happen: The new schedule only takes effect from the time of the update How to reproduce it (as minimally and precisely as possible):
$ TZ=UTC date
Wed May 2 16:32:01 UTC 2018
$ kubectl run test --image=alpine --schedule='33 16 * * *' --restart=Never echo hi
cronjob "test" created
$ kubectl get cronjob
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
test 33 16 * * * False 1 Wed, 02 May 2018 12:33:00 -0400
$ TZ=UTC date
Wed May 2 16:34:08 UTC 2018
$ kubectl get -o yaml cronjob test|sed 's/33 16/34 16/g'|kubectl replace -f -
cronjob "test" replaced
$ kubectl get cronjob
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
test 34 16 * * * False 2 Wed, 02 May 2018 12:34:00 -0400
Anything else we need to know?: Looks like the culprit is at https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/cronjob/utils.go#L113, where the earliestTime to consider scheduling a job is set to the cronjob’s creationTimestamp, without taking the updateTimestamp into account Environment:
- Kubernetes version (use
kubectl version):
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:34:11Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration: GKE
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 43
- Comments: 39 (11 by maintainers)
I think we should assign this bug with the higher priority because this bug causes the unintended job execution with destructive change such as data modification, deletion or some kinds of notification to users. Some may think the jobs should be idempotent, however cronjobs are generally used for repetitive small tasks and making all of such tasks idempotent is the excessive requirement.
I suggest to clear
lastScheduleTimeof the cronjob status with logging the proper message in the case of manually updates.+1 for this issue.
Is there a workaround for this? Is manually deleting and recreating the cronjob the current best method? Triggering jobs immediately on edit makes this unsafe for some uses.
Also ran into this just now, caused an unintentional trigger. changed a cronjob with the purpose of delaying it and caused an immediate execution.
just a +1 for a fix to this issue.
I also just encountered this
I encountered this too, would deleting the job and creating a new one every time we want to update the schedule be the suggested workaround till the issue is fixed?
Making the schedule immutable is not practical, imho.
Please see the new controller being worked on here - https://github.com/kubernetes/kubernetes/pull/93370 - please help test that new controller!
/close
Please, can this be addressed? This happened today:
Usecase: Advising a local dev team that they can edit the schedule of a CronJob to later and prevent it running at the previously specified time; in this case it runs a job to shut down AWS infrastructure overnight in a non production environment.
Outcome: Infrastructure needed to be kept up for longer is immediately terminated!!!
another +1 for more resources on this issue, last year I commented here when we had this happen with a not so important cronjob that cleans tests data from staging env. a few weeks ago we had it with a cronjob that trigger customer emails!
Well, we had a fun morning with this issue. Was meaning to push a cronjob forward for later, but we sent out 1000 emails before we could kill it. Not what you would expect for a simple change of schedule.
We’ve just had a case of a job triggered from cronjob on apply, even when cronjob schedule has not changed.
lastScheduleTimewas set and it was not changed when this unexpected triggering happened.I’m also having this issue on 1.18.9. I just change schedule from 9 UTC to to 10 UTC at 10:50 UTC and gets job execution, that caused some mess in my application. I think this should not happens by default.
Does setting
startingDeadlineSecondsmitigate this issue (docs)? I.e. if the starting deadline is tight enough…For example, I would normally set the
startingDeadlineSecondsto something like a few minutes since I don’t want my CronJobs to run at some random time later if they fail to start. In this case I would expect a change in the schedule to trigger a new Job only if a new Job was supposed to happen somewhere betweenstartingDeadlineSecondsbefore and now (when I change the schedule).It seems that changing
suspend: truetosuspend: falsealso causes immediate job execution. So you have to wait until just after the scheduled time to safely unsuspend.We too ran into this problem this week, several times before we caught on to what was going on.
Still an issue
This is still a problem. /remove-lifecycle rotten