instance-manager: Cluster Autoscaler unable to scale up nodes from EKS autoscaling group warm pool when pods request ephemeral storage

Is this a BUG REPORT or FEATURE REQUEST?:

A bug report.

What happened:

Cluster Autoscaler is unable to scale up nodes from warm pools of an InstanceGroup with the eks provisioner when the unscheduled pods request additional ephemeral storage:

I0215 16:21:32.873817       1 klogx.go:86] Pod nodegroup-runners/iris-kdr5j-mhntj is unschedulable
I0215 16:21:32.873855       1 scale_up.go:376] Upcoming 1 nodes
I0215 16:21:32.873910       1 scale_up.go:300] Pod iris-kdr5j-mhntj can't be scheduled on eks-EksClusterNodegroupDefaultM-lckreA32Rf3D-a6bf610c-e30e-48d5-e342-47ed2155eac7, predicate checking error: node(s) didn't match Pod's node affinity/selector; predicateName=NodeAffinity; reasons: node(s) didn't match Pod's node affinity/selector; debugInfo=
I0215 16:21:32.873939       1 scale_up.go:449] No pod can fit to eks-EksClusterNodegroupDefaultM-lckreA32Rf3D-a6bf610c-e30e-48d5-e342-47ed2155eac7
I0215 16:21:32.873989       1 scale_up.go:300] Pod iris-kdr5j-mhntj can't be scheduled on tm-github-runners-cluster-nodegroup-runners-iris, predicate checking error: Insufficient ephemeral-storage; predicateName=NodeResourcesFit; reasons: Insufficient ephemeral-storage; debugInfo=
I0215 16:21:32.874023       1 scale_up.go:449] No pod can fit to tm-github-runners-cluster-nodegroup-runners-iris
I0215 16:21:32.874050       1 scale_up.go:453] No expansion options

This is the configured InstanceGroup in question:

apiVersion: instancemgr.keikoproj.io/v1alpha1
kind: InstanceGroup
metadata:
  name: iris
  namespace: nodegroup-runners
  annotations:
   instancemgr.keikoproj.io/cluster-autoscaler-enabled: 'true'
spec:
  strategy:
    type: rollingUpdate
    rollingUpdate:
      maxUnavailable: 5
  provisioner: eks
  eks:
    minSize: 1
    maxSize: 10
    warmPool:
      minSize: 0
      maxSize: 10
    configuration:
      labels:
        workload: runners
      keyPairName: iris-github-runner-keypair
      clusterName: tm-github-runners-cluster
      image: ami-0778893a848813e52
      instanceType: c6i.2xlarge

I’m attempting to deploy pods via this RunnerDeployment from the actions-runner-controller. If I remove the ephemeral-storage requests & limits, then Cluster Autoscaler is able to scale up nodes from the warm pool as expected.

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: iris
  namespace: nodegroup-runners
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: workload
                operator: In
                values:
                - runners
      ephemeral: true
      repository: MyOrg/iris
      labels:
        - self-hosted
      dockerEnabled: false
      image: ghcr.io/my-org/self-hosted-runners/iris:v7
      imagePullSecrets:
        - name: github-container-registry
      containers:
        - name: runner
          imagePullPolicy: IfNotPresent
          env:
            - name: RUNNER_FEATURE_FLAG_EPHEMERAL
              value: "true"
          resources:
            requests:
              cpu: "1.0"
              memory: "2Gi"
              ephemeral-storage: "10Gi"
            limits:
              cpu: "2.0"
              memory: "4Gi"
              ephemeral-storage: "10Gi"

What you expected to happen:

For Cluster Autoscaler to be able to scale-in nodes from the warm pool when there’s unscheduled pods that request ephemeral storage.

How to reproduce it (as minimally and precisely as possible):

1. Deploy the above instance group (using your own subnet, cluster, security groups) into a cluster with Cluster Autoscaler in it

2. Deploy any pods with the appropriate node affinity and request limits for ephemeral storage.

3. Check cluster autoscaler logs and note the failure to scale up

Environment:

• Kubernetes version: 1.21

$ kubectl version -o yaml
clientVersion:
  buildDate: "2021-08-19T15:45:37Z"
  compiler: gc
  gitCommit: 632ed300f2c34f6d6d15ca4cef3d3c7073412212
  gitTreeState: clean
  gitVersion: v1.22.1
  goVersion: go1.16.7
  major: "1"
  minor: "22"
  platform: darwin/amd64
serverVersion:
  buildDate: "2021-10-29T23:32:16Z"
  compiler: gc
  gitCommit: 5236faf39f1b7a7dabea8df12726f25608131aa9
  gitTreeState: clean
  gitVersion: v1.21.5-eks-bc4871b
  goVersion: go1.16.8
  major: "1"
  minor: 21+
  platform: linux/amd64

Other debugging information (if applicable):

None available as we’ve deprovisioned our test setup for now but if need be we can reproduce and post additional logs here.