boto3: Sagemaker autoscaling not working using boto3 client

Describe the bug I want to programmatically configure auto-scaling in AWS Sagemaker. I am using the official boto3 documentation. First step is to register a scalable target. Second step is to define a scaling policy on the target. The first step works fine. Second step gets executed but it wont change the metric value .

Steps to reproduce

#Step - 1  -- This step works fine. Max and Min capacities are updated as passed in the arguments
        autoscaling.register_scalable_target(
            ServiceNamespace='sagemaker',
            ResourceId='endpoint/test-20450427-0648/variant/test-20450427-0648-model-kp-vrjfgs268gfljnhs5ra',
            ScalableDimension='sagemaker:variant:DesiredInstanceCount',
            MinCapacity=1,
            MaxCapacity=10,
            RoleARN='arn:aws:iam::654763378725:role/ml-sagemaker'
        )

#Step-2  -- This step is not working. 
        autoscaling.put_scaling_policy(
            PolicyName='InvocationsPerInstanceCustom',
            ServiceNamespace='sagemaker',
            ResourceId='endpoint/test-20450427-0648/variant/test-20450427-0648-model-kp-vrjfgs268gfljnhs5ra',
            ScalableDimension='sagemaker:variant:DesiredInstanceCount',
            PolicyType='TargetTrackingScaling',
            TargetTrackingScalingPolicyConfiguration={
                "TargetValue": 122.0,
                "PredefinedMetricSpecification": {"PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"},
                'ScaleOutCooldown': 60,
                'ScaleInCooldown': 60
            }
        )

Expected behavior Step 2 should update the target value of metric SageMakerVariantInvocationsPerInstance to 122. But it remains empty when checked on AWS console . ScaleOutCooldown and ScaleInCooldown also should be set to 60 but they show default value 300. Debug logs No error message is repoted.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 26 (14 by maintainers)

Most upvoted comments

@georgebakas

PolicyName="SagemakerEndpointInvocationScalingPolicy ", …there is a space here^

pradoz on Aug 10, 2022

Hi @singh0777,

Glad you were able to get it working! I don’t think you did anything wrong here— it seems like, in theory, you should be able to apply custom policies to a target regardless of whether or not a ‘built-in’ policy has already been applied. This may be a bug with the Sagemaker console; I’ve asked the team to elaborate on what the expected behavior should be and am currently waiting on a response.

I have a hunch that your custom policy was actually being applied to the target, but it was not displaying in the console due to the name. I’ll let you know as soon as I have more information!

stobrien89 on May 11, 2021

Hi @stobrien89 . Using SagemakerEndpointInvocationScalingPolicy instead of InvocationsPerInstanceCustom worked. I can see the values updated now. My bad, I thought I can choose any custom policy name and that the updates are done on the basis of metric type SageMakerVariantInvocationsPerInstance. However, I still wonder why boto3 didn’t through any error!.

Thank for your time and help. : )

singh0777 on May 10, 2021

Hi again @singh0777,

One more thing I noticed is that the name of the policy you attached a photo of (from the console) is SagemakerEndpointInvocationScalingPolicy. Is this the same policy you refer to in your code as InvocationsPerInstanceCustom?

stobrien89 on May 10, 2021

Hi @singh0777,

I was able to touch base with a member of the autoscaling team and they confirmed their legacy endpoint was the same as the one provided in the stack trace from your previous boto3 version, so we’ve officially ruled that out at this point.

They mentioned that their best guess for what’s happening here (depending on how these operations are being executed in your code) is a race condition with the RegisterScalableTarget and PutScalingPolicy calls trying to make updates simultaneously.

They suggested you try the following:

describe-scaling-policies via the CLI for the one that’s showing the wrong values after the update does the wrong thing

Try making the RegisterScalableTarget and PutScalingPolicy calls separately (as a test)

Let me know how it goes either way!

stobrien89 on May 7, 2021

Hi @singh0777,

After reviewing your code compared to the documentation, I’m not seeing anything that stands out as problematic. Would you be able to provide a full stack trace for the put_scaling_policy operation by adding boto3.set_stream_logger('') to your code? Please obscure any sensitive information, such as account numbers. Thanks!

stobrien89 on Apr 28, 2021