ray: [Bug] Ray Autoscaler is not spinning down idle nodes due to secondary object copies

Hmm do we know what created those objects and what is referencing them? “ray memory” can show you more information on this.

Yes, thank you @mwtian and @iycheng that worked! At least it worked fine in my single worker node repro scenario above, so hopefully, it will work in general.

I was able to do the “pip install” for the workers using the “setupCommands:” and I was able to the “pip install” for the head here:

  headStartRayCommands:
    - ray stop
    - pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-2.0.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl
    - ulimit -n 65536; ray start --head --no-monitor --dashboard-host 0.0.0.0 &> /tmp/raylogs
    ```

Don’t report memory usage of secondary objects for the purposes of autoscaler downscaling.

I vote for this approach.

Still looking at this. Will have a fix soon.

My experimental run last night with this change looked great, with the Ray Autoscaler scaling down as desired. Thank you very much!!

I did not restart the Ray Cluster after upgrading. But “ray --version” showed the expected version version 2.0.0.dev0 after upgrading on both the head and the worker.

In a k8s deployment, the autoscaler is running in a separate operator node; that node shows that it is running version 2.0.0.dev0 as well.

I can run the pip install in the setupCommands for the workers, but unfortunately, Ray does not support specifying setupCommands for the head when deploying onto K8s. But what I think I can do is bring up the Ray head (with 0 workers), update the head, and then restart the head, if you think that may work.

I’d recommend if possible running head and worker images with the changes built-in rather than running pip install at runtime or in setup commands.

diff --git a/deploy/components/example_cluster.yaml b/deploy/components/example_cluster.yaml
index 1513e8fde..d4359fb77 100644
--- a/deploy/components/example_cluster.yaml
+++ b/deploy/components/example_cluster.yaml
@@ -4,14 +4,14 @@ metadata:
   name: example-cluster
 spec:
   # The maximum number of workers nodes to launch in addition to the head node.
-  maxWorkers: 3
+  maxWorkers: 9
   # The autoscaler will scale up the cluster faster with higher upscaling speed.
   # E.g., if the task requires adding more nodes then autoscaler will gradually
   # scale up the cluster in chunks of upscaling_speed*currently_running_nodes.
   # This number should be > 0.
   upscalingSpeed: 1.0
   # If a node is idle for this many minutes, it will be removed.
-  idleTimeoutMinutes: 5
+  idleTimeoutMinutes: 3
   # Specify the pod type for the ray head node (as configured below).
   headPodType: head-node
   # Optionally, configure ports for the Ray head service.
@@ -35,6 +35,8 @@ spec:
       metadata:
         # The operator automatically prepends the cluster name to this field.
         generateName: ray-head-
+        annotations:
+          node.elotl.co/instance-family-exclusions: "g3,g3s"
       spec:
         restartPolicy: Never
 
@@ -48,11 +50,25 @@ spec:
         containers:
         - name: ray-node
           imagePullPolicy: Always
-          image: rayproject/ray:latest
+          # image: rayproject/ray:latest
+          image: ludwigai/ludwig-ray-gpu:tf-legacy
           # Do not change this command - it keeps the pod alive until it is
           # explicitly killed.
           command: ["/bin/bash", "-c", "--"]
           args: ["trap : TERM INT; touch /tmp/raylogs; tail -f /tmp/raylogs; sleep infinity & wait;"]
+          env:
+          - name: AWS_ACCESS_KEY_ID
+            value: "<censored>"
+          - name: AWS_SECRET_ACCESS_KEY
+            value: "<censored>"
+          - name: AWS_DEFAULT_REGION
+            value: "us-west-2"
+          - name: TUNE_TRIAL_STARTUP_GRACE_PERIOD
+            value: "120.0"
+          - name: TUNE_TRIAL_RESULT_WAIT_TIME_S
+            value: "120"
+          - name: TUNE_STATE_REFRESH_PERIOD
+            value: "5"
           ports:
           - containerPort: 6379  # Redis port
           - containerPort: 10001  # Used by Ray Client
@@ -67,9 +83,9 @@ spec:
             name: dshm
           resources:
             requests:
-              cpu: 1000m
-              memory: 512Mi
-              ephemeral-storage: 1Gi
+              cpu: 7
+              memory: 50Gi
+              ephemeral-storage: 64Gi
             limits:
               # The maximum memory that this pod is allowed to use. The
               # limit will be detected by ray and split to use 10% for
@@ -78,21 +94,25 @@ spec:
               # the object store size is not set manually, ray will
               # allocate a very large object store in each pod that may
               # cause problems for other pods.
-              memory: 512Mi
+              memory: 50Gi
   - name: worker-node
     # Minimum number of Ray workers of this Pod type.
-    minWorkers: 2
+    minWorkers: 0
     # Maximum number of Ray workers of this Pod type. Takes precedence over minWorkers.
-    maxWorkers: 3
+    maxWorkers: 9
     # User-specified custom resources for use by Ray.
     # (Ray detects CPU and GPU from pod spec resource requests and limits, so no need to fill those here.)
     rayResources: {"example-resource-a": 1, "example-resource-b": 1}
+    setupCommands:
+    - pip install boto3==1.17.106 awscli==1.19.106 botocore==1.20.106 s3fs==2021.10.0 aiobotocore==1.4.2 s3transfer==0.4.0 fsspec==2021.10.0
     podConfig:
       apiVersion: v1
       kind: Pod
       metadata:
         # The operator automatically prepends the cluster name to this field.
         generateName: ray-worker-
+        annotations:
+          node.elotl.co/instance-family-exclusions: "g3,g3s"
       spec:
         restartPolicy: Never
         volumes:
@@ -102,9 +122,23 @@ spec:
         containers:
         - name: ray-node
           imagePullPolicy: Always
-          image: rayproject/ray:latest
+          # image: rayproject/ray:latest
+          image: ludwigai/ludwig-ray-gpu:tf-legacy
           command: ["/bin/bash", "-c", "--"]
           args: ["trap : TERM INT; touch /tmp/raylogs; tail -f /tmp/raylogs; sleep infinity & wait;"]
+          env:
+          - name: AWS_ACCESS_KEY_ID
+            value: "<censored>"
+          - name: AWS_SECRET_ACCESS_KEY
+            value: "<censored>"
+          - name: AWS_DEFAULT_REGION
+            value: "us-west-2"
+          - name: TUNE_TRIAL_STARTUP_GRACE_PERIOD
+            value: "120.0"
+          - name: TUNE_TRIAL_RESULT_WAIT_TIME_S
+            value: "120"
+          - name: TUNE_STATE_REFRESH_PERIOD
+            value: "5"
           # This volume allocates shared memory for Ray to use for its plasma
           # object store. If you do not provide this, Ray will fall back to
           # /tmp which cause slowdowns if is not a shared memory volume.
@@ -113,9 +147,9 @@ spec:
             name: dshm
           resources:
             requests:
-              cpu: 1000m
-              memory: 512Mi
-              ephemeral-storage: 1Gi
+              cpu: 7
+              memory: 28Gi
+              ephemeral-storage: 64Gi
             limits:
               # The maximum memory that this pod is allowed to use. The
               # limit will be detected by ray and split to use 10% for
@@ -124,7 +158,8 @@ spec:
               # the object store size is not set manually, ray will
               # allocate a very large object store in each pod that may
               # cause problems for other pods.
-              memory: 512Mi
+              memory: 28Gi
+              nvidia.com/gpu: 1 # requesting 1 GPU
   # Commands to start Ray on the head node. You don't need to change this.
   # Note dashboard-host is set to 0.0.0.0 so that Kubernetes can port forward.
   headStartRayCommands: