controller-runtime: [bug] StorageError: invalid object

Hi All,

We are seeing a lot of error logs when running our operator into production. And we can’t figure out what is happening. The basic flow of what we do goes as follows:

func (r *SparkReconciler) createSparkSubmitterPod(ctx context.Context, spark *runtimev1.Spark) error {
	var pod corev1.Pod
	err := r.Get(ctx, client.ObjectKey{
		Name:      submitterPodName(spark),
		Namespace: spark.Namespace,
	}, &pod)
	if apierrors.IsNotFound(err) {
		r.Log.V(1).Info("Creating the spark submitter pod")
		return r.doCreateSparkSubmitterPod(ctx, spark)
	}
	return errors.Wrap(err, "Failed getting the pod")
}

As you can see in the code snippet. We create a pod only when it does not exist. Because we read from cache we know that the get call can return not found when the pod exists. We do the following in the doCreateSparkSubmitterPod:

if err := r.Client.Create(ctx, pod); err != nil {
    if apierrors.IsAlreadyExists(err) {
	return nil
    }
    return errors.Wrap(err, "Failed creating the pod")
}

So we capture an already exists error. However we tend to see a lot of errors like this when running into production:

"error":"Operation cannot be fulfilled on pods \"255f70af-5699-46c6-8002-a4df45af5209\": StorageError: invalid object, Code: 4, Key: /registry/pods/addatatest2/255f70af-5699-46c6-8002-a4df45af5209, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: f8104e60-1a9d-40ca-847e-52bc0f556844, UID in object meta: "

We have no clue on how to proceed on this. I think it might happen when the pod already exists. But not entirely sure. Perhaps someone can shed a light on this?