gardenlinux: K8s Cluster Nodes with GL 318.4 getting stuck/NotReady

What happened: Running a node with GL 318.4 often gets stuck (as in becoming NotReady) when a certain amount of idling pods are running on it. Nodes with GL 184.0.0 and the same load/amount-of-pods don’t exhibit this behaviour.

When a node becomes NotReady we couldn’t ssh into the node either and existing console session were stalling. From K8s point of view we noticed various events, most prominently Kubelet stopped posting node status, but sometimes also from some “kernel-monitor”:

Events:
  Type     Reason    Age    From            Message
  ----     ------    ----   ----            -------
  Warning  TaskHung  5m23s  kernel-monitor  INFO: task kswapd0:49 blocked for more than 120 seconds.
  Warning  TaskHung  5m22s  kernel-monitor  INFO: task jbd2/sda4-8:363 blocked for more than 121 seconds.
  Warning  TaskHung  5m21s  kernel-monitor  INFO: task dockerd:6500 blocked for more than 121 seconds.
  Warning  TaskHung  5m19s  kernel-monitor  INFO: task dockerd:9030 blocked for more than 122 seconds.
  Warning  TaskHung  5m17s  kernel-monitor  INFO: task dockerd:9551 blocked for more than 122 seconds.
  Warning  TaskHung  5m15s  kernel-monitor  INFO: task containerd:7378 blocked for more than 122 seconds.
  Warning  TaskHung  5m12s  kernel-monitor  INFO: task containerd:8414 blocked for more than 123 seconds.
  Warning  TaskHung  5m9s   kernel-monitor  INFO: task kubelet:3979 blocked for more than 123 seconds.
  Warning  TaskHung  5m6s   kernel-monitor  INFO: task kubelet:4211 blocked for more than 124 seconds.
  Warning  TaskHung  5m5s   kernel-monitor  (combined from similar events): INFO: task kubelet:6670 blocked for more than 124 seconds.

What you expected to happen: GL 318.4 to not perform significantly worse than GL 184

How to reproduce it (as minimally and precisely as possible):

• Create a gardener cluster with two worker pools, one with 318.4 and one with 184.
• Deploy a simple Deployment with anti-affinity:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: some-load
spec:
  replicas: 1
  selector:
    matchLabels:
      name: load
  template:
    metadata:
      labels:
        name: load
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - topologyKey: "kubernetes.io/hostname"
          # preferredDuringSchedulingIgnoredDuringExecution:
          # - podAffinityTerm:
          #     topologyKey: "kubernetes.io/hostname"
          #   weight: 100
      containers:
      - name: load
        image: nicolaka/netshoot
        command:
        - bash
        args:
          - -c
          - |
            sleep 3153600000
        resources:
          limits:
            cpu: 1000m
            memory: 100Mi
          requests:
            cpu: 10m
            memory: 10Mi
      terminationGracePeriodSeconds: 1

• Scale the deployment, I could see kubelet/node flapping with about 70 pods on each worker pool (140 pods in total), e.g. k scale deployment some-load --replicas 140

cc @vpnachev