kubernetes: Deployment status does not match ReplicaSet or Pods

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened: A Deployment’s Status does not match with what the corresponding ReplicaSet/Pods are showing.

What you expected to happen: I would expect to see the same results for both.

Deployment status, notice only 2 ready replicas:

kubectl get deploy myapp -o yaml
...
...
status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: 2018-06-19T20:27:06Z
    lastUpdateTime: 2018-06-20T19:48:02Z
    message: ReplicaSet "myapp-57fd8fcd99" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: 2018-07-26T02:45:13Z
    lastUpdateTime: 2018-07-26T02:45:13Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 4
  readyReplicas: 2
  replicas: 3
  unavailableReplicas: 1

ReplicaSet:

kubectl get rs myapp-57fd8fcd99 -o yaml
...
...
status:
  availableReplicas: 3
  fullyLabeledReplicas: 3
  observedGeneration: 3
  readyReplicas: 3
  replicas: 3

Pods:

kubectl get po -l 'app=myapp'
NAME                                      READY     STATUS    RESTARTS   AGE
myapp-57fd8fcd99-75g2c   1/1       Running   0          10d
myapp-57fd8fcd99-bfwtb   1/1       Running   2          69d
myapp-57fd8fcd99-lql8n   1/1       Running   4          69d

(real application name was replaced with “myapp”)

How to reproduce it (as minimally and precisely as possible): Unfortunately, this is something that started to happen unpredictably ever since our cluster was upgraded to v1.11

Anything else we need to know?: I ran through the CHANGELOG from v1.10 -> v1.11 and these are the changes that may be relevant: https://github.com/kubernetes/kubernetes/pull/61367 https://github.com/kubernetes/kubernetes/pull/61419 https://github.com/kubernetes/kubernetes/pull/61615

Another thing to note is that the deployment wasn’t updated since we last upgraded the cluster but 1 of the pods were recreated after the upgrade.

Environment:

• Kubernetes version (use kubectl version): v1.11.1

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="14.04.5 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.5 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

• Kernel (e.g. uname -a):
• Install tools:
• Others: