kombu: [Bug] AWS Connection.py Using Private Botocore `get_response` Which Expects JSON But is Being Passed XML

Hello Kombu folks!

tl;dr

Kombu is using an internal/private method, get_response, from Botocore which changed behaviour recently due to other Botocore updates to the wire protocol used for SQS, which breaks Kombu. The change has been temporarily reverted, but Kombu cannot depend on a stable behaviour of this method going forward as the changes will soon be applied again.

Full Context/Explanation

The implementation of the AWS asynchronous connection is currently using the non-public method get_response from botocore, here is an example: https://github.com/celery/kombu/blob/3e098dc94ed2a389276ccf3606a0ded3da157d72/kombu/asynchronous/aws/connection.py#L232-L236

get_response is vendored in here: https://github.com/celery/kombu/blob/3e098dc94ed2a389276ccf3606a0ded3da157d72/kombu/asynchronous/aws/ext.py#L1-L9

Botocore recently switched to JSON for the wire protocol it uses to communicate with SQS here (Note: Commit url doesn’t expand nicely, you’ll need to follow the link).

Kombu builds XML AWS Query API requests manually (instead of using something like the boto3 SDK client) to communicate with SQS: https://github.com/celery/kombu/blob/3e098dc94ed2a389276ccf3606a0ded3da157d72/kombu/asynchronous/aws/connection.py#L185-L203

The resulting XML responses are then passed to the aforementioned get_response, which now expects JSON due to the change in Botocore also linked above, for parsing. In this situation, get_response returns a response object with an empty body, which Kombu reads to be a legitimate empty result. This means that a request to get messages, for example, will always return empty. In this way it fails silently.

Botocore has reverted the change here. But this is only a temporary fix, as JSON is the preferred wire protocol.

Possible Paths Forward

1. Kombu could move to using the public boto3 SDK to interact with SQS (create queues, read messages, etc) instead of hand crafting AWS Query API requests. pros: Seamless support from a public interface which will handle the wire protocol migration smoothly cons: This is a large refactoring, and presumably, there is some historical reason that we’re unaware of for why Kombu is hand crafting requests since it is a complex piece of code and there was likely a reason for the effort to build it this way.
2. Re-institute the XML parsing that was previously in place before the migration to get_response (which was done in this PR here). pros: Smaller scope for the change which could likely be completed sooner. Code worked in the past cons: More complex handcrafted request/response handling existing in Kombu rather than simply using boto3 SDK
3. Something else entirely? Please let us know what you think!

Thanks in advance for your time 🙏

CC:

• MWAA: @kytha, @rafidka, @dhegberg, @yumike2006
• Airflow: @onikolas, @vincbeck
• Boto: @nateprewitt