boto3: DatabaseMigrationService.Waiter.ReplicationTaskStopped doesn't work for recently started jobs

Use case

I have a job that kicks off a DMs replication task on a periodic basis.

Expected behaviour

DatabaseMigrationService.Waiter.ReplicationTaskStopped waits for the replication task to reach a stopped or error state.

Actual behaviour

Errors out immediately with:

Traceback (most recent call last):
  File "lambda-dms-kickoff.py", line 41, in <module>
    lambda_handler({},{})
  File "lambda-dms-kickoff.py", line 33, in lambda_handler
    'Values': [DMS_TASK_ARN]
  File "/Users/Jared/git-repos/aws-lambda-attachment-extractor/.venv/lib/python3.6/site-packages/botocore/waiter.py", line 53, in wait
    Waiter.wait(self, **kwargs)
  File "/Users/Jared/git-repos/aws-lambda-attachment-extractor/.venv/lib/python3.6/site-packages/botocore/waiter.py", line 323, in wait
    last_response=response,
botocore.exceptions.WaiterError: Waiter ReplicationTaskStopped failed: Waiter encountered a terminal failure state

This is also the case with the AWS CLI:

aws --region us-east-1 dms wait replication-task-stopped --filters "Name=replication-task-arn,Values=$replication_task_arn"

Waiter ReplicationTaskStopped failed: Waiter encountered a terminal failure state

Although if I check the status manually, I can see it’s in a non-terminal and non-failure state:

aws --region us-east-1 dms describe-replication-tasks --filters "Name=replication-task-arn,Values=$replication_task_arn" --query "ReplicationTasks[0].Status"

"starting"

Code

    start_resp = dms.start_replication_task(
        ReplicationTaskArn=DMS_TASK_ARN,
        StartReplicationTaskType='reload-target'
    )

    time.sleep(3)

    waiter = dms.get_waiter('replication_task_stopped')
    waiter.wait(Filters=[
        {
            'Name': 'replication-task-arn',
            'Values': [DMS_TASK_ARN]
        },
    ])

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 6
  • Comments: 18 (2 by maintainers)

Most upvoted comments

This is still an issue which makes waiters for ‘running’ and ‘stopped’ states almost unusable. Is there any plans to fix this?

@swetashre Can we re-open this issue?

Currently facing similar issue with replication_task_stopped. Would be glad if someone can work on this as it’s required to complete a client’s solution of automating data transfer from source database to on-premise. We are using python and hence boto3

@DeepanshKhurana you could certainly do it that way as well. I wasn’t necessarily trying to replace the waiter configs exactly.

I edited my code to explain what those two variables are:

    WAIT_TIME = 15 # seconds to wait between each poll
    MAX_RETRY = 600 # the max number of seconds to poll before timing out