kubernetes: Exceeded quota for pod priority class

I am able to reproduce this on a 1.21.alpha cluster, and it definitely seems to be a timing issue / race condition.

Below is my all-in-one script to reproduce it, and an excerpt of the output to follow. It took about 15 minutes for the problem to occur.

#!/bin/bash
date

while true
do
  kubectl apply -f - << EOF
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: pvg-k8s-pod-priority
value: 10
EOF

  kubectl apply -f - << EOF
apiVersion: v1
kind: ResourceQuota
metadata:
  name: pvg-k8s-pod-priority
spec:
  hard:
    pods: "2"
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values:
      - pvg-k8s-pod-priority
EOF

  kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx1
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  priorityClassName: pvg-k8s-pod-priority
EOF

  kubectl wait pod nginx1 --for=condition=ready

  kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx2
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  priorityClassName: pvg-k8s-pod-priority
EOF

  echo "Expecting an error while creating pod #3 using this priority class..."
  kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx3
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  priorityClassName: pvg-k8s-pod-priority
EOF

  if [ $? -eq 0 ]; then 
    echo "WHOOPS, I HAVE 3 PODS RUNNING IN THIS PRIORITY CLASS!!!"
    date
    exit 1
  fi

  kubectl wait pod nginx2 --for=condition=ready

  kubectl delete pod {nginx1,nginx2}
  kubectl delete ResourceQuota pvg-k8s-pod-priority
  kubectl delete pc pvg-k8s-pod-priority

  echo 
done

An excerpt of this script’s output when it happened:

... Lots of iterations omitted ...

priorityclass.scheduling.k8s.io/pvg-k8s-pod-priority created
resourcequota/pvg-k8s-pod-priority created
pod/nginx1 created
pod/nginx1 condition met
pod/nginx2 created
Expecting an error while creating pod #3 using this priority class...
Error from server (Forbidden): error when creating "STDIN": pods "nginx3" is forbidden: exceeded quota: pvg-k8s-pod-priority, requested: pods=1, used: pods=2, limited: pods=2
pod/nginx2 condition met
pod "nginx1" deleted
pod "nginx2" deleted
resourcequota "pvg-k8s-pod-priority" deleted
priorityclass.scheduling.k8s.io "pvg-k8s-pod-priority" deleted

priorityclass.scheduling.k8s.io/pvg-k8s-pod-priority created
resourcequota/pvg-k8s-pod-priority created
pod/nginx1 created
pod/nginx1 condition met
pod/nginx2 created
Expecting an error while creating pod #3 using this priority class...
pod/nginx3 created
WHOOPS, I HAVE 3 PODS RUNNING IN THIS PRIORITY CLASS!!!
Thu 11 Mar 2021 08:15:33 AM EST

The quota knows it has more pods than allowed:

~/projects/ktest $ kubectl get quota pvg-k8s-pod-priority
NAME                   AGE   REQUEST     LIMIT
pvg-k8s-pod-priority   17m   pods: 3/2  

All 3 pods are running:

~/projects/ktest $ kubectl get pod
NAME     READY   STATUS    RESTARTS   AGE
nginx1   1/1     Running   0          15m
nginx2   1/1     Running   0          15m
nginx3   1/1     Running   0          15m

All 3 pods have the priority class:

~/projects/ktest $ k describe pod nginx1 | grep priority
Priority Class Name:  pvg-k8s-pod-priority
~/projects/ktest $ k describe pod nginx2 | grep priority
Priority Class Name:  pvg-k8s-pod-priority
~/projects/ktest $ k describe pod nginx3 | grep priority
Priority Class Name:  pvg-k8s-pod-priority

My version:

~/projects/ktest $ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.0-alpha.3-438-g16b909ce148325.dev-1614052066", GitCommit:"16b909ce1483258464715c9ab76b4dff82d596e5", GitTreeState:"clean", BuildDate:"2021-02-23T03:48:38Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

@brianpursley I was finally able to recreate the problem as well but it took much longer for me.

Recreate on 1.20:

priorityclass.scheduling.k8s.io/pvg-k8s-pod-priority created
resourcequota/pvg-k8s-pod-priority created
pod/nginx1 created
pod/nginx1 condition met
pod/nginx2 created
Expecting an error while creating pod #3 using this priority class...
pod/nginx3 created
WHOOPS, I HAVE 3 PODS RUNNING IN THIS PRIORITY CLASS!!!
Fri Mar 12 04:14:53 UTC 2021

Recreate on 1.18:

priorityclass.scheduling.k8s.io/pvg-k8s-pod-priority created
resourcequota/pvg-k8s-pod-priority created
pod/nginx1 created
pod/nginx1 condition met
pod/nginx2 created
Expecting an error while creating pod #3 using this priority class...
pod/nginx3 created
WHOOPS, I HAVE 3 PODS RUNNING IN THIS PRIORITY CLASS!!!
Fri Mar 12 04:03:45 UTC 2021

we took a look to this one, will wait until next triage to see if the author could reproduce it again as suggested.