botocore: Error when creating SQS queues with moto

Describe the bug

With latest version of botocore (1.29.127), my unit tests started to fail when creating a fake SQS queue with moto.

It works fine with 1.29.126 though so I suspect https://github.com/boto/botocore/commit/3e1f77d8c8eae589cdfb4b9ec09c816570f19904 could be the cause of it.

Expected Behavior

The fake queue should be created as expected.

Current Behavior

aws_resource_sqs.create_queue(QueueName="OurFakeQueue") returns an empty dictionary rather than a Queue object.

Reproduction Steps

import moto
import boto3
from uuid import uuid4

@moto.mock_sqs
def test_create_queue():
    sqs = boto3.resource("sqs", region_name="us-east-1")

    q_name = str(uuid4())[0:6]
    new_queue = sqs.create_queue(QueueName=q_name)

    print(new_queue)

test_create_queue()

With pip install boto3==1.26.126 botocore==1.29.126 moto, the output is sqs.Queue(url='https://sqs.us-east-1.amazonaws.com/123456789012/a94c6c')

With pip install boto3==1.26.127 botocore==1.29.127 moto, the output is {}

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.29.127

Environment details (OS name and version, etc.)

Darwin lmi-macbookpro183.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64

About this issue

Original URL
State: closed
Created a year ago
Reactions: 7
Comments: 16 (7 by maintainers)

Most upvoted comments

Please note: changes to the SQS protocol were reverted in botocore 1.29.129

kellertk on May 8, 2023

For everyone else, the issue is recorded for the developers of moto in https://github.com/getmoto/moto/issues/6286 and is currently being looked at. While we wait for a resolution there, the short term solution if you’re using moto is going to be to pin both boto3 and botocore.

Downgrading boto3 by itself won’t work because every minor version release will always use the latest version of botocore. We wouldn’t recommend pinning to boto3<=1.25.

boto3<1.26.127
botocore<1.29.127

nateprewitt on May 5, 2023

We getting the same error, but we use client. This error can be reproduced with this code:

import boto3
import moto

with moto.mock_sqs():
    sqs = boto3.client("sqs", region_name="eu-west-2")
    create_resp = sqs.create_queue(QueueName="test")
    print("QueueUrl" in create_resp)

We expect QueueUrl property to be in response as in documentation (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs/client/create_queue.html#SQS.Client.create_queue)

Versions:

Python 3.10.10 botocore 1.29.127 moto 4.1.6 boto3 1.26.126

dikiigr on May 5, 2023

Sorry, I was trying to clarify that the issue is indeed moto, AWS requests to production are fine.

Ealameda31 on May 5, 2023

Yes, this issue doesn’t even have to do with moto After the update of botocore to 1.29.127, create_queue is not returning “QueueUrl” as it should

Could you clarify this a bit more @joaomaranhao? Are you saying live calls to SQS (not using moto) are no longer returning QueueUrl?

I was just saying that the method is not returning “QueueUrl” as documentation says But actually that call was made with moto! Sorry about that!

joaomaranhao on May 5, 2023

This issue was remediated for future releases of Boto3 in https://github.com/getmoto/moto/pull/6331. We’ll resolve this now that there’s a path forward, thanks everyone for your patience.

nateprewitt on Jun 2, 2023

I was just saying that the method is not returning “QueueUrl” as documentation says

Sorry I keep asking the same question but want to make sure we’re being exact with the issue. Is this with or without moto? If it’s with moto, that’s entirely expected with the current breakage.

Edit; I saw your edit, thank you!

nateprewitt on May 5, 2023

@nateprewitt internally AWS url in queues are now request-based and not queue-based.

So instead of sqs.Queue(url=“https://sqs.us-west-2.amazonaws.com/123456789012/{queue_name}”) you get something like sqs.Queue(url={request_uuid}).

moto might be expecting the full url to the queue … because the request is made an has a QueueUrl in the params:

api_params = {‘QueueUrl’: ‘XXXXXXXXXXXXXXXX’}.

@Ealameda31 we’re aware that what’s currently in moto won’t function correctly with the new format. A lot of pieces are going to be broken.

What I do want to get a conclusive answer on though is are you seeing functional differences with calls to the production Amazon SQS service (no mocking)? If that’s the case, I’d be very interested to get more details on reproduction. If you can open a new ticket if that’s the case and we can start tracking it there.

nateprewitt on May 5, 2023