ray: [Bug] [Autoscaler] Autoscaler can start unnecessary nodes due to poor bin packing heuristic ordering

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Clusters

What happened + What you expected to happen

When running the repro script underneath that creates

1 placement group with 31 CPUs + 1 GPU

head node: 16 CPUs worker nodes 1 GPU + 16 CPUS

we are supposed to get 1 head node + 1 gpu node.

This is the case when running Ray 1.6, but in Ray 1.7, it starts one more CPU node unnecessary.

1.6 ray status

Node status
---------------------------------------------------------------
Healthy:
 1 head_node
 1 gpu_node
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

1.7 ray status

Node status
---------------------------------------------------------------
Healthy:
 1 head_node
 1 gpu_node
 1 cpu_node
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Reproduction script

Script

import time

import ray
from ray.util.placement_group import (
    placement_group,
    placement_group_table,
    remove_placement_group
)


@ray.remote(num_cpus=1)
class Worker(object):
    def __init__(self, i):
        self.i = i

    def work(self):
        time.sleep(0.1)
        print("work ", self.i)


@ray.remote(num_cpus=1, num_gpus=1)
class Trainer(object):
    def __init__(self, i):
        self.i = i

    def train(self):
        time.sleep(0.2)
        print("train ", self.i)


def main():
    ray.init(address="auto")
             # cluster_env="jun-julia-working-2:1")

    bundles = [{"CPU": 1, "GPU": 1}]
    bundles += [{"CPU": 1} for _ in range(30)]

    pg = placement_group(bundles, strategy="PACK")

    ray.get(pg.ready())

    workers = [Worker.options(placement_group=pg).remote(i) for i in range(30)]

    trainer = Trainer.options(placement_group=pg).remote(0)

    while True:
        ray.get([workers[i].work.remote() for i in range(30)])
        ray.get(trainer.train.remote())



if __name__ == "__main__":
    main()

Env

base_image: "anyscale/ray-ml:pinned-nightly-py37"
debian_packages: []

python:
  pip_packages: []
  conda_packages: []

post_build_cmds:
  - pip uninstall -y ray
  # - pip install -U https://ray-wheels.s3.us-west-2.amazonaws.com/releases/1.7.0/2367a2cb9033913b68b1230316496ae273c25b54/ray-1.7.0-cp37-cp37m-manylinux2014_x86_64.whl
  # 1.6
  # - pip install -U ray==1.6
  # master
  - pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-2.0.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl
  - pip3 install -U ray[default]

Cluster

cloud_id: {{env["ANYSCALE_CLOUD_ID"]}}
region: us-west-2

aws:
    BlockDeviceMappings:
        - DeviceName: /dev/sda1
          Ebs:
            VolumeSize: 500

head_node_type:
    name: head_node
    instance_type: m5.4xlarge

worker_node_types:
    - name: cpu_node
      instance_type: m5.4xlarge
      min_workers: 0
      max_workers: 10
      use_spot: false
    - name: gpu_node
      instance_type: g3.4xlarge
      min_workers: 0 
      max_workers: 10
      use_spot: false

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!