kubernetes: Server-side apply: migration from client-side apply leaves stuck fields in the object

Migrating from client-side apply to server-side apply (as currently implemented in kubectl and documented here) leaves the objects in a somewhat corrupted state, and result in the resources forever being inconsistent with their expected state. Namely, removed fields that have been set in the initial client-side applied version stay in the resource forever and aren’t dropped.

What happened:

If I apply a manifest using client-side apply and then switch to server-side apply with no changes, the operation succeeds. However, if I then delete a field from the manifest and run server-side apply again, the field stays in the resource instead of being removed

What you expected to happen:

I would obviously expect the field to be removed from the resource.

How to reproduce it (as minimally and precisely as possible):

1. Create a configmap in the cluster with client-side apply:

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: test
   data:
     key: value
     legacy: unused
   EOF
   

2. Confirm that the configmap is fine:

   $ kubectl get configmap -o yaml test | egrep -A 3 '^data:'
   data:
     key: value
     legacy: unused
   kind: ConfigMap
   

3. Apply the same manifest with server-side apply:

   cat <<EOF | kubectl apply --server-side -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: test
   data:
     key: value
     legacy: unused
   EOF
   

4. Confirm that it wasn’t changed:

   $ kubectl get configmap -o yaml test | egrep -A 3 '^data:'
   data:
     key: value
     legacy: unused
   kind: ConfigMap
   

5. Remove one of the values from the configmap and apply again using server-side apply:

   cat <<EOF | kubectl apply --server-side -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: test
   data:
     key: value
   EOF
   

6. Check the configmap, expecting the legacy value to be removed:

   $ kubectl get configmap -o yaml test | egrep -A 3 '^data:'
   data:
     key: value
     legacy: unused
   kind: ConfigMap
   

7. As you can see, the legacy value is still there.

Anything else we need to know?:

This happens because the client-side apply results are recorded in managedFields with an Update operation by the kubectl-client-side-apply field manager. When the resource is migrated to server-side apply, the results are instead tracked as Apply by the kubectl field manager. If there are no conflicts between the two fields (and naturally you wouldn’t expect them to be there, since you’re trying to convert the same resource to SSA), both field managers will own the fields and the client-side apply won’t get kicked out. If you then remove a field from the list of fields managed by SSA, the field will still be managed by the old Update from kubectl-client-side-apply and wouldn’t be removed as a result.

Environment:

• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T21:51:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}