kubernetes: Many ConfigMaps and Pods slow down cluster, until it becomes unavailable (since 1.12)

What happened:

I schedule multiple jobs in my cluster. Each job uses a different ConfigMap which contains the configuration for that job.

This worked well on version 1.11 of Kubernetes. After upgrading to 1.12 or 1.13, I’ve noticed that doing this will cause the cluster to significantly slow down; up to the point where nodes are being marked as NotReady and no new work is being scheduled.

For example, consider a scenario in which I schedule 400 jobs, each with its own ConfigMap, which print “Hello World” on a single-node cluster would.

On v1.11, it takes about 10 minutes for the cluster to process all jobs. New jobs can be scheduled. On v1.12 and v1.13, it takes about 60 minutes for the cluster to process all jobs. After this, no new jobs can be scheduled.

What you expected to happen:

I did not expect this scenario to cause my nodes to become unavailable in Kubernetes 1.12 and 1.13, and would have expected the behavior which I observe in 1.11.

How to reproduce it (as minimally and precisely as possible):

The easiest way seems to be to schedule, on a single-node cluster, about 300 jobs:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: job-%JOB_ID%
data:
# Just some sample data
  game.properties: |
    enemies=aliens
---
apiVersion: batch/v1
kind: Job
metadata:
  name: job-%JOB_ID%
spec:
  template:
    spec:
      containers:
      - name: busybox
        image: busybox
        command: [ "/bin/echo" ]
        args: [ "Hello, World!" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
      volumes:
        - name: config-volume
          configMap:
            name: job-%JOB_ID%
      restartPolicy: Never
  backoffLimit: 4

I can consistently reproduce this issue in a VM-based environment, which I configure using Vagrant. You can find the full setup here: https://github.com/qmfrederik/k8s-job-repro

Anything else we need to know?:

Happy to provide further information as needed

Environment:

• Kubernetes version (use kubectl version): v1.12 through v1.13
• Cloud provider or hardware configuration: bare metal
• OS (e.g: cat /etc/os-release): 18.04.1 LTS (Bionic Beaver)
• Kernel (e.g. uname -a): Linux vagrant 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kubeadm
• Others: