karmada: Modifying the Job's PropogationPolicy causes the Job to rebuild unexpectedly

What happened:

We had a batch of jobs, and the original PropagationPolicy is configured as clusterA and clusterB. Jobs have been successfully created in both clusters, and only executed in the clusterA cluster.

# get jobs on clusterA  
$ kubectl get job 
NAME                        COMPLETIONS   DURATION   AGE
test-new-cronjob-27940858   1/1           26s        6m4s
test-new-cronjob-27940859   1/1           26s        5m4s
test-new-cronjob-27940860   1/1           27s        4m4s
test-new-cronjob-27940861   1/1           26s        3m4s
test-new-cronjob-27940862   1/1           26s        2m4s
test-new-cronjob-27940863   1/1           27s        64s
test-new-cronjob-27940864   0/1           4s         4s

# get jobs on clusterA  
$ kubectl get job 
NAME                        COMPLETIONS   DURATION   AGE
test-new-cronjob-27940858   0/0           0s         6m3s
test-new-cronjob-27940859   0/0           0s         5m3s
test-new-cronjob-27940860   0/0           0s         4m3s
test-new-cronjob-27940861   0/0           0s         3m3s
test-new-cronjob-27940862   0/0           0s         2m3s
test-new-cronjob-27940863   0/0           0s         63s
test-new-cronjob-27940864   0/0           0s         3s

Then we adjusted pp configuration, remove clusterA, keep only clusterB. We found that the FULLYAPPLIED of all jobs’ ResourceBinding(including jobs that have been successfully delivered but not cleared) changed to False

$ kubectl --kubeconfig=karmada.kubeconfig get rb
NAME                            SCHEDULED   FULLYAPPLIED   AGE
test-new-cronjob-27940858-job   True        False          6m20s
test-new-cronjob-27940859-job   True        False          5m20s
test-new-cronjob-27940860-job   True        False          4m20s
test-new-cronjob-27940861-job   True        False          3m20s
test-new-cronjob-27940862-job   True        False          2m20s
test-new-cronjob-27940863-job   True        False          80s
test-new-cronjob-27940864-job   True        False          20

The reason for false is that the k8s control plane of clusterB rejected the request of karmada-agent to modify the job configuration. At the same time, the old jobs ran in clusterA had been deleted.

status:
  aggregatedStatus:
  - appliedMessage: 'Failed to apply all manifests (0/1): Job.batch "test-new-cronjob-27940860"
      is invalid: spec.completions: Invalid value: 1: field is immutable'
    clusterName: clusterB
  conditions:
  - lastTransitionTime: "2023-02-15T09:04:16Z"
    message: Failed to apply all works, see status.aggregatedStatus for details
    reason: FullyAppliedFailed
    status: "False"
    type: FullyApplied
  - lastTransitionTime: "2023-02-15T09:00:01Z"
    message: Binding has been scheduled
    reason: BindingScheduled
    status: "True"
    type: Scheduled

Then we adjusted the pp configuration and add clusterA back

View the rb resources on the control plane, and you can find that the rb resources left over from the failure of the second step of tuning have been retuned successfully, and the job has also been recreated and delivered to clusterA for execution again.

$ kubectl --kubeconfig=karmada.kubeconfig get rb 
NAME                            SCHEDULED   FULLYAPPLIED   AGE
test-new-cronjob-27940858-job   True        True           24m
test-new-cronjob-27940859-job   True        True           23m
test-new-cronjob-27940860-job   True        True           22m
test-new-cronjob-27940861-job   True        True           21m
test-new-cronjob-27940862-job   True        True           20m
test-new-cronjob-27940863-job   True        True           19m
test-new-cronjob-27940864-job   True        True           18m

What you expected to happen:

I think this is a bug need to be fixed. When the status of the job changes to execution completion (completed or failure), these jobs should not be rescheduled after the pp change, otherwise it will cause unexpected repeated execution of the job.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Karmada version: 1.2.2
• kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version):
• Others: