kubernetes: kubectl apply doesn't remove container args

I’m running into an oddity where, as the title says, an update to a Deployment is not fully applying the new PodTemplateSpec onto the previous version, as components of the previous version remain around after the update.

I’m running v1.2.0-alpha8 on CoreOS 835.9.0

Here’s an example of creating a Deployment (v1), and applying an update (v2) which simply removes certain parts of the original spec (the args key in this case), to showcase that the new spec isn’t fully being applied, even though the Deployment’s last-applied-configuration says it should have.

git-deployment-v1.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: git-deployment
  labels:
    name: git-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: git-deployment
    spec:
      containers:
        - name: git-sync
          image: gcr.io/google_containers/git-sync
          imagePullPolicy: Always
          args:
            - -rev=origin/gh-pages
          volumeMounts:
            - name: markdown
              mountPath: /git
              readOnly: false
          env:
            - name: GIT_SYNC_REPO
              value: https://github.com/kubernetes/kubernetes
            - name: GIT_SYNC_DEST
              value: /git
            - name: GIT_SYNC_WAIT
              value: "120"
      volumes:
        - name: markdown
          emptyDir: {}

We create the Deployment:

kubectl create -f git-deployment-v1.yaml

When the Pod is running, it should not have any annotations regarding the Deployment as it was the initial deployment, and there are no history of updates. We can verify that it is not set by seeing that this cmd returns: <no value>

kubectl get deployment git-deployment -o go-template='{{index .metadata.annotations "kubectl.kubernetes.io/last-applied-configuration"}}'

…and we can check that the args key was set, in this case, to pass an arg of -rev=origin/gh-pages to the container’s entry cmd, by checking:

kubectl get po git-deployment-<ID> -o yaml | grep -A 1 "args"

it returns:

 - args:
  - -rev=origin/gh-pages

…which is what we expect as we set it as such in the v1 Deployment

---

After making a change in the PodTemplateSpec, specifically, removing the args key all-together (this is the only change made in the spec), we have a v2 Deployment to apply as an update to v1:

git-deployment-v2.yaml:

 apiVersion: extensions/v1beta1
 kind: Deployment
 metadata:
   name: git-deployment
   labels:
     name: git-deployment
 spec:
   replicas: 1
   template:
     metadata:
       labels:
         name: git-deployment
     spec:
       containers:
         - name: git-sync
           image: gcr.io/google_containers/git-sync
           imagePullPolicy: Always
           volumeMounts:
             - name: markdown
               mountPath: /git
               readOnly: false
           env:
             - name: GIT_SYNC_REPO
               value: https://github.com/kubernetes/kubernetes
             - name: GIT_SYNC_DEST
               value: /git
             - name: GIT_SYNC_WAIT
               value: "60"
       volumes:
         - name: markdown
           emptyDir: {}

We apply the update:

kubectl apply -f git-deployment-v2.yaml

…and we can verify that the new PodSpecTemplate for the v2 Deployment is what we wanted it to be by examining the output of the last-applied-configuration key for the newly updated Pod:

kubectl get deployment git-deployment -o go-template='{{index .metadata.annotations "kubectl.kubernetes.io/last-applied-configuration"}}' | python -m json.tool

(this output and the git-deployment-v2.yaml manifest should match, and neither should have the args key as we specified)

---

However, if you examine the new Pod’s args again using

kubectl get po git-deployment-<NEW_ID> -o yaml | grep -A 1 "args"

…it still returns:

 - args:
  - -rev=origin/gh-pages

…which should not be the case 👎

So in my case, its still passing a flag via the args key, which in v2 I determined I did not want by removing it, and the existing flag is messing with the updated Pod’s execution.

Any ideas what could be taking place? Thanks!